Skip to content
This repository has been archived by the owner on Jan 4, 2018. It is now read-only.

Load historical data into BigQuery #1

Open
mimming opened this issue May 9, 2016 · 1 comment
Open

Load historical data into BigQuery #1

mimming opened this issue May 9, 2016 · 1 comment
Assignees

Comments

@mimming
Copy link

mimming commented May 9, 2016

Load up the historical data, currently living in that big file in Google Cloud Storage

  • define the schema
  • Scrub the data
    • unify timestamps
    • environment
      • accelerometer -> camera
      • temperature (without accelerometer) -> computer
    • weather
      • this one is pretty clean
  • Check the schemas into here (in case we need to recreate tables later)
  • Get it into BigQuery
@mimming mimming self-assigned this May 18, 2016
@mimming
Copy link
Author

mimming commented May 18, 2016

A lot of it is up at https://bigquery.cloud.google.com/dataset/panoptes-survey:unit_sensors

but, the schema of the old data does not match the new data, and about 200k of the weather records didn't load.

I'll take another swing at getting all the data loaded, and into the same schema (so we can do big aggregate queries across all the data)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant