bdre planning roadmap

SL No.	Category	Task	Description	Owner	Status	Priority	Target Date	Comments
	Quality	Test existing features	Testing of navigation,UI etc.
	Quality	Test existing Jobs	Create different types of workflows from BDRE (e.g, DQ,Hive Test Data Generation,Ingestion,etc.
	Quality	Testing existing features	black box testing of API
	Testing	Write missing test cases	Write missing Junit test cases.
	Quality	Implement and enforce Sonar	Bring back Sonar testing and code quality check
	Strategic	Make BDRE open source
	Feature	Develop UI for workflow creation
	Feature	Deployment automation
	Feature	Flume integration and file ingestion
	Feature	Code upload feature from BDRE
	Feature	Enhancement of D2P ,feature with tarball export
	Feature	Data Ingestion from an API end point periodically
	Infrastructure	BDRE inhouse cluster
	Infrastructure	Dedicated end networking for BDRE
	Feature	Data Profiling	collect cardinalities of several fields as the data is being read to process and supplement it with the Data Model feature
	Feature	Integrate Spark with BDRE	Develop Java wrapper class to invoke Spark from Oozie or invoke spark submit via a shell script.Real value is in reusing existing spark context for an application as generally spark context initialization for every job on Yarn is very slow. This would also allow us to reuse RDDs across jobs reusing the same context.Make changes in the workflow.
	Feature	Workflow validation at design time	Is it possible to inspect and validate compatibility of tasks defined in a workflow at design time? Can we make the components to adhere to a standard interface via which they would advertise what type of data they would be expecting which would help do the validation?
	Feature	Performance Monitoring	Build a statistical model capturing the performance profile of a job and flag any violations
	Feature	Decision support for Oozie workflow
		Build end-to-end usecases for BDRE
	Marketing	Video for BDRE demo
	Feature	Implementing landing dashboard for BDRE
	Feature	Hive to RDBMS data export
		Onboard user experience expert
	Marketing	Marketable presentation for BDRE
	Marketing	Press release for BDRE
	Feature	File level lineage for BDRE
	Documentation	BDRE technical documentation
	Documentation	BDRE functional documentation
	Strategic	Build engineering friendly environment.