1 ''' B{BigJob Documentation}
2
3 This is the BigJob implementation of the Pilot API. Pilot-Jobs (also referred to as
4 Pilot Compute) decouple system level and application level job management.
5
6 The main concepts and classes exposed by the Compute part of the API are:
7
8 - L{PilotCompute} (PC):
9 a pilot job, which can execute some compute workload (L{ComputeUnit}).
10
11 - L{PilotComputeDescription} (PCD):
12 description for specifying the requirements of a \L{PilotCompute}.
13
14 - L{PilotComputeService} (PCS):
15 a factory for creating \L{PilotCompute}s.
16
17 The data side of the Pilot API is symmetric to the compute side. The exposed
18 classes for managing Pilot Data are:
19
20 - L{PilotData} (PD):
21 a pilot that manages some data workload (L{DataUnit})
22
23 - L{PilotDataDescription} (PDD):
24 a abstract description of the requirements of the PD
25
26 - L{PilotDataService} (PDS):
27 a factory (service) which can create L{PilotData}s according to some
28 specification
29
30
31 The application workload is represented by so called L{ComputeUnit}s and L{DataUnit}s:
32
33 - L{ComputeUnit} (CU):
34 a work item executed on a L{PilotCompute}.
35
36
37 - L{DataUnit} (DU):
38 a data item managed by a L{PilotData}
39
40 Both Compute and Data Units are specified using an abstract description object:
41
42 - L{ComputeUnitDescription} (CUD):
43 abstract description of a L{ComputeUnit}.
44
45 - L{DataUnitDescription} (DUD):
46 abstract description of a L{DataUnit}.
47
48
49 The L{ComputeDataService} represents the central entry point for the application
50 workload:
51
52 - L{ComputeDataService} (CDS)
53 a service which can map CUs and DUs to a set of Pilot Compute and Pilot Data.
54
55 The L{ComputeDataService} (CDS) takes care of the placement of Compute and Data Units.
56 The set of L{PilotCompute}s and L{PilotData} available to the CDS can be changed during
57 the application's runtime. The CDS different data-compute affinity and will handle
58 compute/data co-locationing for the requested data-compute workload.
59
60 Pilots, Compute and Data Units are associated with a L{State}.
61
62 - L{State<pilot.api.compute.api.State>}: State enumeration
63
64
65 B{Example}::
66
67 from pilot import PilotComputeService, ComputeDataService, State
68
69 pilot_compute_service = PilotComputeService()
70
71 # create pilot job service and initiate a pilot job
72 pilot_compute_description = {
73 "service_url": 'fork://localhost',
74 "number_of_processes": 1,
75 "working_directory": os.path.join(os.getcwd(),"work"),
76 'affinity_datacenter_label': "eu-de-south",
77 'affinity_machine_label': "mymachine"
78 }
79
80 pilotjob = pilot_compute_service.create_pilot(pilot_compute_description=pilot_compute_description)
81
82 compute_data_service = ComputeDataService()
83 compute_data_service.add_pilot_compute_service(pilot_compute_service)
84
85 # start work unit
86 compute_unit_description = {
87 "executable": "/bin/date",
88 "arguments": [""],
89 "number_of_processes": 1,
90 "output": "stdout.txt",
91 "error": "stderr.txt",
92 "affinity_datacenter_label": "eu-de-south",
93 "affinity_machine_label": "mymachine"
94 }
95
96 compute_unit = compute_data_service.submit_compute_unit(compute_unit_description)
97
98 compute_data_service.wait()
99
100 compute_data_service.cancel()
101
102
103 All API objects that should be utilized by the application reside in the L{pilot} namespace. The implementation resides in L{pilot.impl}.
104
105 Please, refer to U{https://github.com/saga-project/BigJob/tree/master/examples/pilot-api} for an extensive set of examples.
106 '''
107
108
109 application_id = "bigjob"
110
111
112 from pilot.impl.pilotcompute_manager import PilotComputeService
113 from pilot.impl.pilotcompute_manager import PilotCompute
114 from pilot.impl.pilotcompute_manager import ComputeUnit
115 from pilot.impl.pilotdata_manager import PilotDataService
116 from pilot.impl.pilotdata_manager import PilotData
117 from pilot.impl.pilotdata_manager import DataUnit
118 from pilot.impl.pilot_manager import ComputeDataService, ComputeUnitService, DataUnitService
119 from pilot.impl.pilot_manager_decentral import ComputeDataServiceDecentral
120 from pilot.api.api import PilotError
121 from pilot.api import State
122
123 import bigjob
124
125 """ Version of Pilot-API/BigJob """
126 version = bigjob.version
127