AWS Data Wrangler 1.1.0

New Functionalities

• Support for nested arrays and structs on wr.s3.to_parquet() #206
• Support for Read Parquet/Athena/Redshift chunked by number of rows #192
• Add custom_classifications to wr.emr.create_cluster() #193
• Support for Docker on EMR #193
• Add kms_key_id, max_file_size, region arguments to wr.db.unload_redshift() #197
• Add catalog_versioning argument to wr.s3.to_csv() and wr.s3.to_parquet() #198
• Add keep_files and ctas_temp_table_name arguments to wr.athena.read_sql_*() #203
• Add replace_filenames argument to wr.s3.copy_objects() #215

Enhancements

• wr.s3.to_csv() and wr.s3.to_parquet() no longer need delete table permission to overwrite catalog table #198
• Added support for UUID on wr.db.read_sql_query()(PostgreSQL) #200
• Refactoring of Athena encryption and workgroup support #212

Bug Fix

• Support for read full NULL columns from PostgreSQL, MySQL, and Redshift #218

Thanks

We thank the following contributors/users for their work on this release:

@robkano ,@luigift, @parasml, @OElesin, @jar-no1, @keatmin, @pmleveque, @sapientderek, @jadayn, @igorborgest.

---

P.S. Lambda Layer's zip-file and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 1.0.4

New Functionalities

• Add wr.s3.copy_objects and wr.s3.merge_datasets #186
• Registering module's type annotations #194

Enhancements

• Support append mode for wr.catalog.create_parquet_table and wr.catalog.create_csv_table #188

Docs

• Adding a note about collisions for wr.catalog.sanitize_dataframe_columns_names #185

Thanks

We thank the following contributors/users for their work on this release:

@JPFrancoia, @deathrowe, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 1.0.3

New Functionalities

• Add moto support for S3 and EMR (partially) #109

Enhancements

• Add CSV tutorials #181

Bug Fix

• Fix cast for char and varchar lengths #182
• Fix Athena issues with boto3 session #179

Thanks

We thank the following contributors/users for their work on this release:

@russellbrooks, @vincentclaes, @JPFrancoia, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 1.0.2

New Functionalities

• Add new wr.catalog.extract_athena_types() #141

Enhancements

• Add validate_schema to wr.s3.to_parquet() #167

Bug Fix

• Add CSV Dataset utilities to wr.s3.to_csv #170
• Fix CSV decompression #175
• Fix missing boto3_session #172

Thanks

We thank the following contributors/users for their work on this release:

@vfrank66, @JPFrancoia, @jewelltp, @hjuhel-cdpq, @jar-no1, @rmlove, @josecw, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 1.0.1

New Functionalities

• categories arg in s3.read_parquet, db.unload_redshift, athena.read_sql_query [#160]

Enhancements

• Athena's table and columns names sanitisation revisited [#161]

Bug Fix

• Add support for Athena queries on workgroups without encryption [#159]

Thanks

We thank the following contributors/users for their work on this release:

@vfrank66, @nitin-kakkar, @sapientderek, @nagomiso, @igorborgest.

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 1.0.0

1.0.0 🎉

Check out the brand new documentation page!

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 0.3.2

New Functionalities

• Add header and filename arguments to Pandas.to_csv()

Enhancements

• Pandas.read_parquet() will return Int64 for integers with null values mixed #132
• Pandas.to_redshift() now is able to cast Int64 for integers with null values mixed #132

Bug Fixies

• s3.head_object_with_retry() public again #133

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 0.3.1

New Functionalities

• Add pandas.read_fwf(), read_fwf_list(), read_fwf_prefix() for fixed-width files #131
• Support for compressed files for pandas.read_csv(), read_csv_list() and read_csv_prefix() #129
• Support for consistent view on emr.create_cluste() #130

Enhancements

• Support for Python 3.8
• Bumping Pandas version to 1.0.1
• Bumping PyArrow version to 0.16.0

Docs

• New documentation page

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 0.3.0

Enhancements

• Support for Pandas 1.0.0
• Support for all pandas.read_csv() arguments
• Support for custom VARCHAR length for Aurora and Redshift

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).

AWS Data Wrangler 0.2.6

Enhancements

• Smaller Lambda layers #113
• Support for categorical partitions for Pandas.to_parquet() #115
• Support for RangeIndex for Pandas.to_parquet() #111
• Add columns parameter for Pandas.to_csv() #110
• Add columns parameter for Pandas.to_aurora() #110
• Improving NaN handling during Pandas.read_sql_athena()
• Small performance improvements

Bugfixes

• Fixing bug to unload null values from Aurora #114

---

P.S. Lambda Layer's bundle and Glue's wheel/egg are available below. Just upload it and run!

P.P.S. Have you never used Layers? Check the step-by-step guide.

P.P.P.S. AWS Data Wrangler counts on compiled dependencies (C/C++) so there is no support for Glue PySpark by now (Only Glue Python Shell).