performance, file size issue

[pavlexander]

Hi!

Just started using this library and trying to compare it's performance to main competitor Parquet.NET (which is also referenced on a main page)!

the data

    public class SingleCandlestickMP
    {
        public long Id { get; set; }
        public long OpenTime { get; set; }
        public long Open { get; set; }
        public long High { get; set; }
        public long Low { get; set; }
        public long Close { get; set; }
        public long Volume { get; set; }
        public long CloseTime { get; set; }
    }

2_673_685 number of rows

parquetSharp data save approaches

I have so far managed to save the data using 2 approaches that seemingly yield the same results.

approach 1:

        public static async Task SaveToParquet2(this IEnumerable<SingleCandlestickMP> data, string fileFullPath)
        {
            var columns = new Column[]
            {
                new Column<long>("Id"),
                new Column<long>("OpenTime"),
                new Column<long>("Open"),
                new Column<long>("High"),
                new Column<long>("Low"),
                new Column<long>("Close"),
                new Column<long>("Volume"),
                new Column<long>("CloseTime")
            };

            var rows = data.ToArray();

            using var writer = ParquetFile.CreateRowWriter<SingleCandlestickMP>(fileFullPath, columns, Compression.Snappy);
            writer.WriteRows(rows);
            writer.Close();
        }

approach 2

        public static async Task SaveToParquet2(this IEnumerable<SingleCandlestickMP> data, string fileFullPath)
        {
            var columns = new Column[]
            {
                new Column<long>("Id"),
                new Column<long>("OpenTime"),
                new Column<long>("Open"),
                new Column<long>("High"),
                new Column<long>("Low"),
                new Column<long>("Close"),
                new Column<long>("Volume"),
                new Column<long>("CloseTime")
            };

            var i = data.Select(x => x.Id).ToArray();
            var ot = data.Select(x => x.OpenTime).ToArray();
            var o = data.Select(x => x.Open).ToArray();
            var h = data.Select(x => x.High).ToArray();
            var l = data.Select(x => x.Low).ToArray();
            var c = data.Select(x => x.Close).ToArray();
            var v = data.Select(x => x.Volume).ToArray();
            var ct = data.Select(x => x.CloseTime).ToArray();*/

        using (var stream = new FileStream(fileFullPath, FileMode.Create))
        {
            using var writer = new ManagedOutputStream(stream);
            using var fileWriter = new ParquetFileWriter(writer, columns);

            using RowGroupWriter rowGroup = fileWriter.AppendRowGroup();

            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(i);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(ot);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(o);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(h);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(l);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(c);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(v);
            }
            using (var objectIdWriter = rowGroup.NextColumn().LogicalWriter<long>())
            {
                objectIdWriter.WriteBatch(ct);
            }
            }
        }

quick summary and a side question

personally I don't see much use in approach 2, since it's more complex and more error prone. Is there something I gain from it when compared to approach 1?

parquet.net approach

Regardless, moving on back to the main topic. Here is the code I am using to save the data in parquet.net

        public static async Task SaveToParquet(this IEnumerable<SingleCandlestickMP> data, string fileFullPath)
        {
            await ParquetSerializer.SerializeAsync(data, fileFullPath, new ParquetSerializerOptions()
            {
                CompressionMethod = Parquet.CompressionMethod.Snappy
            });
        }

see that in all 3 approaches Snappy compression method is used

results

• time takes to save the file ParquetSharp: ~800ms (817). File size: 81gb (approach 1)
• time takes to save the file Parquet.net: ~800ms (788). File size: 31gb.

results interpretation

As you see both libraries take almost the same amount of time to save the file. This is OK.
But then the main issue is the file size. Is there a reason why ParquetSharp result takes that much space?

I have extracted the metadata with ParquetViewer and compared both files. It seems like there are quite a lot of differences. Some of them are (left: parquet.net, right: parquetSharp):

schema version:

encodings:

something else

package versions

• <PackageReference Include="Parquet.Net" Version="4.16.4" /> (Parquet.Net version 4.16.4 (build 42339e08d7520ef1301b27689e1d1c02d91b058e))
• <PackageReference Include="ParquetSharp" Version="12.0.1" /> (parquet-cpp-arrow version 12.0.1)

[marcin-krystianc]

Hi @pavlexander,

I think the factor to consider is default column encodings used by Parquet.Net and ParquetSharp.
Try disabling dictionary encoding (ParquetSharp) and enabling DeltaBinaryPacked encoding and see whether it improved the storage size:

     var builder = new ParquetSharp.WriterPropertiesBuilder();
     builder.DisableDictionary();
     builder.Encoding(ParquetSharp.Encoding.DeltaBinaryPacked);

[adamreeve]

I also just came across this change that was merged in Arrow recently that is supposed to reduce the encoded size significantly for the delta binary packed encoding in some cases: apache/arrow#37940

It looks like that should hopefully be released in version 14.0

[adamreeve]

I did some quick testing and changing the kValuesPerBlock parameter didn't have much affect on file size, but the fix that was merged in apache/arrow#37940 significantly decreases the size of files in my uniformly distributed test case, and brings the size in line with Parquet.Net. So it looks like once 14.0.0 is released we'll have much improved file sizes when using the delta binary packed encoding.

[pavlexander]

@adamreeve Hey! Thank you for your time checking on this issue! I would like to replicate your testing with arrow v14 on my own dataset. Could you please help me?

edit1: so far I have managed to build the ParquetSharp solution successfully after installing cmake and fixing the path env variable for msbuild. Using Windows.

edit2: the next thing I did is downloaded an arrow source tagged with v14, then did the
cmake .. --preset ninja-release-minimal followed by cmake --build .
the output is arrow.dll, arrow.exp, arrow.lib. I then removed a nuget reference in ParquetSharp project and tried adding a dll assembly dependency, but got an error:

the reference is invalid or unsupported..

I don't really have much experience with interop, so if you could point me on obvious mistake - I would much appreciate it :)

Thank you again for help, regardless

[adamreeve]

Hi @pavlexander, sure, we statically link the ParquetSharp C++ wrapper to the Arrow library so you won't be able to test this by just linking a different dll unfortunately. The way I tested this was copy the Arrow port from vcpkg (https://github.com/microsoft/vcpkg/tree/master/ports/arrow) and update it to point to a recent arrow commit, and use this as a vcpkg "overlay-port" in my local code. I've put my changes to test arrow 14 in a branch on this repository (https://github.com/G-Research/ParquetSharp/tree/arrow-14-test) so you could try rebuilding from that branch. But by pushing it to this repo our CI has built a NuGet package you can download and use for testing which is probably easiest, look for the "nuget-package" link under artifacts at: https://github.com/G-Research/ParquetSharp/actions/runs/6500781100

[pavlexander]

that's perfect! Used the nuget package you provided.

As a reminder - here are the old results:

• Snappy,DeltaBinaryPacked,True Elapsed: 597 ms, Size: 39.25 mb
• Zstd,DeltaBinaryPacked,True Elapsed: 513 ms, Size: 31.45 mb

new results (ParquetSharp 13.0 beta-1 is what I believe the Nuget package version was called)

• Snappy,DeltaBinaryPacked,True Elapsed: 416 ms, Size: 31.04 mb
• Zstd,DeltaBinaryPacked,True Elapsed: 301 ms, Size: 29.99 mb

here is the final comparison of previous and new results

snappy

• (prev) parquet.net: 853 ms, File size: 31.26325035095215 mb
• (prev) parquetSharp: 526 ms, File size: 39.24139404296875 mb
• (new) ParquetSharp: Elapsed: 416 ms, Size: 31.04 mb

zstd

• (prev) Parquet.net: 1106 ms, File size: 27.30514907836914 mb
• (prev) ParquetSharp: 772 ms, File size: 31.449615478515625 mb
• (new) ParquetSharp: Elapsed: 301 ms, Size: 29.99 mb

in conclusion

Snappy compression with the new arrow version is going to be better, just on par with parquet.net. HOWEVER, it looks like the performance is twice as good. This is a huge improvement. Note that I am running apps in debug mode during testing, so the results might differ in release mode and with different datasets.

In regards to Zstd compression I see that parquet.net is slightly more effective, but is it worth 4x processing time? :)

@adamreeve,
I will keep this topic active for a bit and wait until the new version arrives, so I can re-test.. Unless I forget about it then feel free to close it as you wish :) Thank you very much for help, my curiosity has been fulfilled.

@marcin-krystianc much appreciate your help as well!