additional support for decimal type

[nirmal82]

help needed
I am trying to generate parquet file with existing schema having field with following datatype
fixed_len_byte_array(16) LICENSE_TERM_IN_MONTHS (DECIMAL(38,0));

requesting for enabling support for this dataType

[wilwade]

@nirmal82 We only just added Decimal support in v1.2.3 (https://github.com/LibertyDSNP/parquetjs/releases/tag/v1.2.3)

If you upgrade to that release it should work!

If it doesn't, then it sounds like we might have a bug.

[nirmal82]

@wilwade

I am using the latest code. even while reading the parquet it generating some Buffer value instead of number/string. i am not good with javascript, so can you please help me to unblock with reading and writing parquet having DECIMAL datatype

[wilwade]

@nirmal82 could you share an example file that produces this issue?

[nirmal82]

@wilwade attaching my sample parquet file and json data files both

here are the steps

• extracting schema from parquet using reader
• reading content from json file
• passing the schema and content to parquet writer for generating new parquet files.

testParquet.zip

Thanks

[wilwade]

@nirmal82 I see that other readers work with it, but it doesn't appear to be supported.

https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal

Decimal is required to have precision <= 18, but this has it set at 38

pqrs schema decimal.parquet
Metadata for file: decimal.parquet

version: 1
num of rows: 1
created by: parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)
metadata:
  writer.model.name: example
message spark_schema {
  REQUIRED FIXED_LEN_BYTE_ARRAY (16) ID (DECIMAL(38,0));
  OPTIONAL BYTE_ARRAY NAME (STRING);
  OPTIONAL BYTE_ARRAY ADDRESS (STRING);
  OPTIONAL INT96 CREATE_DTM;
  OPTIONAL INT96 UPDATE_DTM;
}

[wilwade]

Which now that I read it again, you said in your original post, I just missed it.

[wilwade]

Looked a bit more into it. So this library currently only supports up to 18 precision because it doesn't convert over that into the correct fixed_len_byte_array or binary base type.

I'm not sure we would be able to do so without just using strings for the input and output due to the limitations of JavaScript working with big decimals.

Open to ideas or PRs for sure!

[wilwade]

@nirmal82 I added a new issue to better document the issue for those searching and track the support: #91