-
Notifications
You must be signed in to change notification settings - Fork 129
bdre planning roadmap
Sri Harsha Boda edited this page Sep 15, 2017
·
1 revision
SL No. | Category | Task | Description | Owner | Status | Priority | Target Date | Comments |
---|---|---|---|---|---|---|---|---|
Quality | Test existing features | Testing of navigation,UI etc. | ||||||
Quality | Test existing Jobs | Create different types of workflows from BDRE (e.g, DQ,Hive Test Data Generation,Ingestion,etc. | ||||||
Quality | Testing existing features | black box testing of API | ||||||
Testing | Write missing test cases | Write missing Junit test cases. | ||||||
Quality | Implement and enforce Sonar | Bring back Sonar testing and code quality check | ||||||
Strategic | Make BDRE open source | |||||||
Feature | Develop UI for workflow creation | |||||||
Feature | Deployment automation | |||||||
Feature | Flume integration and file ingestion | |||||||
Feature | Code upload feature from BDRE | |||||||
Feature | Enhancement of D2P ,feature with tarball export | |||||||
Feature | Data Ingestion from an API end point periodically | |||||||
Infrastructure | BDRE inhouse cluster | |||||||
Infrastructure | Dedicated end networking for BDRE | |||||||
Feature | Data Profiling | collect cardinalities of several fields as the data is being read to process and supplement it with the Data Model feature | ||||||
Feature | Integrate Spark with BDRE | Develop Java wrapper class to invoke Spark from Oozie or invoke spark submit via a shell script.Real value is in reusing existing spark context for an application as generally spark context initialization for every job on Yarn is very slow. This would also allow us to reuse RDDs across jobs reusing the same context.Make changes in the workflow. | ||||||
Feature | Workflow validation at design time | Is it possible to inspect and validate compatibility of tasks defined in a workflow at design time? Can we make the components to adhere to a standard interface via which they would advertise what type of data they would be expecting which would help do the validation? | ||||||
Feature | Performance Monitoring | Build a statistical model capturing the performance profile of a job and flag any violations | ||||||
Feature | Decision support for Oozie workflow | |||||||
Build end-to-end usecases for BDRE | ||||||||
Marketing | Video for BDRE demo | |||||||
Feature | Implementing landing dashboard for BDRE | |||||||
Feature | Hive to RDBMS data export | |||||||
Onboard user experience expert | ||||||||
Marketing | Marketable presentation for BDRE | |||||||
Marketing | Press release for BDRE | |||||||
Feature | File level lineage for BDRE | |||||||
Documentation | BDRE technical documentation | |||||||
Documentation | BDRE functional documentation | |||||||
Strategic | Build engineering friendly environment. |