DataFlow - Consolidate Observation Frames #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This creates a template for a dataflow job. The job accepts a location of an observation sequence as stored in the `panoptes-detected-sources` bucket. For now this is a full bucket path but soon will be just the `sequence_id`. Currently adds all the CSV files together, then filters the sources that don't appear in all the frames. Also computes the normalized data so the similar stars finder can be added. Todo: * Accept just a `sequence_id`. * Figure out best way to call. * Add in the `frameThreshold` param. * Do Similar Stars step
also calculate the Similar Stars. The latter takes a long time to run and could probably be optimized. Also currently giving bad scores. :)
Change frame threshold to 95%
…-make-observation-psc
…s-network into df-make-observation-psc
* Better check for object_id from event bucket.
…s-network into df-make-observation-psc
…s-network into df-make-observation-psc
…ically update the sha256 in the kubernetes file.
…gger a pubsub call that triggers the `gce-find-similar-sources`. Will also be used to update the `sequences` metadata (TODO).
similar source file
Remove db utils
wtgee
added a commit
that referenced
this pull request
May 13, 2019
* Adding RGB endpoint for CR2 files * TODO: Convert FITS; timelapse; jpgs * Removing old dataflow items * Adding build scripts for containers and kuberenets clusters for the similar source finder. * Removing old Cloud SQL connections * Misc cleanup
wtgee
added a commit
that referenced
this pull request
May 16, 2019
* Based off `panoptes-utils` docker image * Upload master observation to bucket. Previously this was being done by #29. * Misc cleanup
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A dataflow job template that, given a
sequence_id, will gather CSV files from thepanoptes-detected-sourcesbucket and consolidate them into one large master PSC collection. This file is then uploaded topanoptes-observation-psc, which will trigger a pubsub message (see the similar source finder readme for details).This job could feasibly either be replaced by simplified cloud function and storage compose utility or be expanded to include the source filtering currently happening as part of the similar source finder.