Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't pull data from s3 bucket using Meltano with the only change between s3 and local being the path #26

Open
Vaslo opened this issue Jun 18, 2022 · 0 comments

Comments

@Vaslo
Copy link

Vaslo commented Jun 18, 2022

Spend quite a few hours on this. I'm trying to pull down the DBT Jaffleshop csv data directly from the s3 folder. These settings keep pulling nothing:

version: 1
default_environment: dev
project_id: faa34bcc-b471-4943-a928-8104b0330bbd
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
        - format: detect
          name: customers
          path: s3://dbt-tutorial-public/
          pattern: "jaffle_shop_customers*.csv"
          start_date: "2017-05-01T00:00:00Z"
          key_properties: [id]

However when I change it to a local path, it very easily pulls it. Is there something else I'm supposed to do with the s3 settings? I didn't see anything else I needed to change.

version: 1
default_environment: dev
project_id: faa34bcc-b471-4943-a928-8104b0330bbd
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
        - format: detect
          name: customers
          path: file:///home/me/dbt_data
          pattern: "jaffle_shop_customers*.csv"
          start_date: "2017-05-01T00:00:00Z"
          key_properties: [id]

I have even forced the format to CSV to be pull, but no luck. Thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant