Post your use cases / software / screenshots here

[aloneguid]

Hi everyone! This is an open thread for posting examples of the way you are using Parquet.Net. I'm looking for real world usage!

For example:

I use Parquet.Net to store and query my collection of memes. It's very fast and efficient, and it supports all kinds of data types, like strings, integers, Booleans, and even emojis! 😂

Of course it's a joke, but you get the idea!

How do you use Parquet.Net? Share your examples below, and don't forget to check you are not violating your employer's privacy policy, intellectual rights and so on. You don't want to end up like Bob, who accidentally leaked the company's secret formula for their best-selling product. He thought he was just sharing a harmless photo of his lunch, but he didn't notice the paper on his desk with the confidential information. Now he is looking for a new job and a new place to eat.

[aloneguid]

ML.NET seems to be using Parquet.Net in order to decode files, reference to .csproj.

[aloneguid]

ParquetViewer - native Windows Forms app.

[mithom]

we use it to store data gathered in testcell tests. It forms the backend of our data sharing/storing and processing framework around testing. When implemented correctly (still WIP), it would allow our calculation engineers to search through all the test results and filter based on some of the data of the test results

[aloneguid]

Recfluence seems to be using parquet.net to read data.

[aloneguid]

Azure Synapse Analytics powershell export script in the official Microsoft repo.

[aloneguid]

Parquet Online is an experimental parquet metadata (in future data) viewer. Works fully offline in browser (compiled to web assembly).

[sbrickey]

we use it as a replacement for Azure Data Factory, which saves us about $10k/month

our design is simple: table to store source/destinations and some tracking flags. Process will check the table, identify which records are "ready", transfers data, then updates the records to indicate their completion... we use this to keep cloud data "current" from the source system(s)

ADF cost was split about half between "bytes transferred" and "API calls" (interactions with the table above)

we replaced it with custom code running on a B-series VM, which costs around $130/month

the custom code is simple: generate local parquet file from source, upload to storage, execute API call into databricks to merge parquet file into existing data.

[immitev]

We use the library for implementing SSIS+ components for reading/writing Parquet files https://cozyroc.com/ssis/parquet .

Well, the initial development is based on an older version of the library, due to requiring .NET Framework 4.0 support (did some backporting). Luckily the upcoming version of our component suite can target .NET Framework 4.6.1, so that we are transitioning to the 3.10.0 version.

[aloneguid]

Cinchoo ETL is an ETL framework for .NET. From website:

Simple, intuitive Extract, transform and load (ETL) library for .NET. Extremely fast, flexible, and easy to use. Cinchoo ETL is a code-based ETL framework for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

seems to be using Parquet.Net for reading and writing parquet files.

[NeilMacMullen]

Used in Kusto-Loco KQL engine for serialisation of local data but, more importantly, as the preferred format for serialising query results over the wire. It's a great library - thank you! :-)