hflow
=====

Hadoop visual workflows built on Oozie framework

The project is on Hadoop Visual Workflows. One can create a visual workflow of map jars , reduce jars, and map+reduce jars and start the hadoop workflows.
Jar files can be uploaded by end-users. We create a Oozie Xml which can be forwarded to a Oozie server. The server can then deploy hadoop sequence of required map-reduce tasks/jobs in the back-end.
We also have an in-house testing functionality in which we created a server on the machine. It listens to curl requests from the browser on a particular port and deploys the workflow in the machine itself.
The UI is simple drag and drop. Jar files are on a scrollable desk on top and can be dropped in the required order to create workflow. A workflow can be modified / deleted by delete ( red-cross ) button.


Particulars :
1. We used html , php, javascript and  ajax to implement the functionalities.
2. The sequence / flow of jobs is stored in /tmp/job ( see job.php ).
3. Input file by default is /tmp/note ( see server.py ) .
4. Final output/result by default is written in /var/www/cloud/result ( see server.py ) .
5. Folders maps/ , reds/ and mapred/ contain the default and uploaded jar files of respective types.
6. Temporary files and folders are created in hdfs, which are deletd on completion of workflow:
	1. /in		-	default input file
	2. /out		- 	default output folder
7. The oozie xml is created in the browser in the div with id='oozie-xml' , which can be used to forward to the server.


Installation :
1. The entire cloud folder has to be placed anywhere in the server accessible path. The changed to be done are :
	1. Path is to the folder has to be specified in '$homeDir' variable in main.php. Ex : In unix, it can be placed in /var/www and $homeDir would be '/var/www/cloud/' ( note the last '/' ). (Note: 'localhost' can be replaced by server IP ).
	2. In 'submit.php' , absolute path has to provided in '$path' , and '$home' will be same as $homeDir of main.php
	3. In server.py , path variable ( line number 11 ) has to be changed.

2. Port number can be changed in job.php and server.py if not free.



Assumptions :
1. The user will only upload working jar files, and in the proper location ( map, reduce or map-reduce jars ) .
2. Workflow is logistically correct. Example : output of one job can be the input of the next job. Oozie server should be looking into errors but our own hadoop backend doesn't , and will throw errors / exceptions . i
3.Hadoop is installed in '/usr/local/'