Jenkins

Jenkins is used to run the User Data processing. The project configuration is fairly simple and should be fairly straightforward.

First, on Jenkins, you need to install the Any Build Step plugin. You can do this through the Manage Plugins area in the Jenkins Configuration. I'd also recommend installing the Rebuild but it's not strictly required.

Once you have that installed, you can create a new freestyle project (New Item > Freestyle Project). The build will have the following settings:

1. The build is parameterized
   • It will take 3 parameters
   • config.json - A file parameter
   • SUCCESS_POST_URL - a string parameter
   • FAILURE_POST_URL - a string parameter
2. Restrict where the project can be run
   • Create a new machine to attach to Jenkins and set it up with everything needed to run the User Data Processing script
   • Attach it to Jenkins as a node that has a label that this job is restricted to
3. Set it up to pull down the git repository User Data Processing
4. Allow triggering builds remotely
5. For the build steps, run the User Data Processing on the config.json file (e.g. python user-data-processor.py config.json)
6. For the post-build actions, add a Flexible Publish step. (You will need to select Any Build Step in the Jenkins configuration screen under Flexible Publish.)
   • Add a conditional step to run when the build is successful that does a shell command to curl $SUCCESS_POST_URL
   • Similarly, add a conditional step to run when the build failed that does curl $FAILURE_POST_URL

Then you can setup your environment variables and turn on processing and it will start processing data. Jenkins will handle queuing jobs and you can add more nodes to run the code if you need to handle more jobs.