A SAGA-based Pilot-Job
BigJob, a SAGA-based Pilot-Job, is a general purpose pilot-job framework. Pilot-Jobs support the use of container jobs with sophisticated workflow management to coordinate the launch and interaction of actual computational tasks within the container. This results in the decoupling of workload submission from resource assignment, allowing a flexible execution model that enables the distributed scale-out of applications on multiple and possibly heterogeneous resources. It allows the execution of jobs without the necessity to queue each individual job.
BigJob has been used to support various execution patterns and execution workflows. SAGA-BigJob has been used to execute scientific applications categorized as embarrassingly parallel applications and loosely coupled applications on scalable distributed resources.
BigJob is an open source project, hosted and developed by the SAGA Project and the RADICAL Group at the Cloud and Autonomic Computing Center at Rutgers University.
BigJob is available via PyPi and installs in seconds with easy_install (Installation instructions)
$ easy_install bigjob
Code Example: Running a number of jobs on local or remote resource using a grid middleware scheduling system