Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support maxLength in JSON Schema #105

Open
techtangents opened this issue Aug 7, 2024 · 2 comments
Open

Support maxLength in JSON Schema #105

techtangents opened this issue Aug 7, 2024 · 2 comments

Comments

@techtangents
Copy link
Contributor

Is your feature request related to a problem? Please describe.
My Redshift warehouse uses this tap, and is experiencing disk size bloat. It appears to be much bigger than I'd expect (2 orders of magnitude). I suspect this is due to target-redshift creating all columns as varchar(10000), despite my source data (Postgresql) having mostly varchar(256).

Redshift docs advise to use the smallest possible column size for data (see here and here).

Describe the solution you'd like
If the input stream supplies a "maxLength" property for a string field, use that as the size of the field created. This behaviour could be turned on via a config setting.

Describe alternatives you've considered
Not sure. Possibly running a bunch of "alter table" queries after ingestion.

Additional context
I would like to use this in conjunction with a tap-postgresql change described at MeltanoLabs/tap-postgres#477

@techtangents
Copy link
Contributor Author

@tobiascadee
Copy link
Contributor

@techtangents you are welcome to open a pr for this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants