You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
My Redshift warehouse uses this tap, and is experiencing disk size bloat. It appears to be much bigger than I'd expect (2 orders of magnitude). I suspect this is due to target-redshift creating all columns as varchar(10000), despite my source data (Postgresql) having mostly varchar(256).
Redshift docs advise to use the smallest possible column size for data (see here and here).
Describe the solution you'd like
If the input stream supplies a "maxLength" property for a string field, use that as the size of the field created. This behaviour could be turned on via a config setting.
Describe alternatives you've considered
Not sure. Possibly running a bunch of "alter table" queries after ingestion.
Additional context
I would like to use this in conjunction with a tap-postgresql change described at MeltanoLabs/tap-postgres#477
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
My Redshift warehouse uses this tap, and is experiencing disk size bloat. It appears to be much bigger than I'd expect (2 orders of magnitude). I suspect this is due to target-redshift creating all columns as varchar(10000), despite my source data (Postgresql) having mostly varchar(256).
Redshift docs advise to use the smallest possible column size for data (see here and here).
Describe the solution you'd like
If the input stream supplies a "maxLength" property for a string field, use that as the size of the field created. This behaviour could be turned on via a config setting.
Describe alternatives you've considered
Not sure. Possibly running a bunch of "alter table" queries after ingestion.
Additional context
I would like to use this in conjunction with a tap-postgresql change described at MeltanoLabs/tap-postgres#477
The text was updated successfully, but these errors were encountered: