No documentation for Config file.

[TebanSierra]

There is no documentation on what the config file MUST have, what is optional and what are the possible values of each input.

[aaronsteers]

Ditto - I landed here and had the same concern. @TebanSierra - By chance did you figure this out and can you share back here in this convo?

[aaronsteers]

It looks like at least these are required:

• account_id - aws account?
• external_id - not sure what this is?
• role_name - iam role, presumably?
• region_name - i.e. us-east-1, us-west-2, etc.

There are probably more but these are the ones explicitly 'required': https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/__init__.py#L15

[aaronsteers]

Not the right platform, I know, but these setup instructions seem helpful for the Dynamo side. I did not find anything similar on AWS's docs site. https://fivetran.com/docs/databases/dynamodb/setup-guide

[aaronsteers]

Updates from my research (and thanks to the great help of others in the community):

1. There's an outstanding PR (Add support for different AWS authentication types #22) which reduces these to only requiring region_name.
2. The tap scans and identifies all tables in the AWS account. The current main branch assumes a certain type of credential inheritance but the Add support for different AWS authentication types #22 branch can inherit from the local environment.
3. There's another PR (Only return streams of tables, where user has read access for. #10) which poses that if the account can see any tables which it doesn't actually have access to, that may fail the process.
4. Meltano #2216 (https://gitlab.com/meltano/meltano/-/issues/2216) discusses that the catalog discovered is of type=object, and it may be desirable (necessary?) to amend/modify the discovered schema in order to declare any specific columns.
   • TODO: I'm not positive if the tap still "works" with the default disovered schema, and if so, what the output would look like. My (very uninformed) assumption would be that it should still extract/load all the correct data, but in a json structure rather than in proper columnar format.
     • If anyone can confirm this default behavior, that would be very much appreciated.
5. Research seems to confirm this list of config options:
   • account_id - Required until Add support for different AWS authentication types #22 merges. (Afterwards AWS creds can be inherited from the environment.)
   • external_id - Ditto. Required until Add support for different AWS authentication types #22 merges. (Afterwards AWS creds can be inherited from the environment.)
   • role_name - Ditto. Required until Add support for different AWS authentication types #22 merges. (Afterwards AWS creds can be inherited from the environment.)
   • region_name - Required. The region from which tables should be scanned/synced.
   • use_local_dynamo - Optional. This looks like a optional testing flag, only applicable when using some local dynamo service emulator. (Not confirmed.)

[shareef]

I just installed tap-dynamodb and trying to extract data out of a table....I have accesskey and secretkey setup in environment variables....how do I extract data. What is external_id?