AWSBatch module

class AWSBatch.batch(infrastructure_type, region='us-east-1', batch_namespace='batch-namespace', queue_name='my-queue-1', computeEnvironmentName='TwinFlow_batch_env', minvCpus=1, maxvCpus=500, instanceTypes='optimal', IAM_role=None)

Bases: object

Handle both the IaC for AWS Batch and utilization of queues, job submission, etc.

The IaC is relying on AWS SDK, in the future it will be swapped for CDK.

Parameters:

  • infrastructure_type (str) – Determines underlying infrastructure type. e.g. EKS, Fargate, EC2

  • infrastructure_type – Options are: ‘ec2’, ‘eks’,’fargate’

  • region (str, optional) – The default is “us-east-1”.

  • batch_namespace (str, optional) – The default is “batch-namespace”.

  • queue_name (str, optional) – The default is “my-queue-1”.

  • computeEnvironmentName (str, optional) – The default is ‘TwinFlow_batch_env’.

  • minvCpus (int, optional) – Any value greater than or equal to zero. A value of zero means the cluster will auto close all compute instances when finished, but requires the longest start up time for new jobs. If a user selects a warm startup configuration such as a value of 500, a cluster will remain provisioned and provide the fastest startup time for new jobs, but also includes costs of keeping these instances open. The default is 1.

  • maxvCpus (int, optional) – Prevent AWS Batch from using more than this many cpus. The default is 500.

  • instanceTypes (str, optional) – Specify the exact instant type or leave this as default for Batch auto selection. When defining batch jobs with ‘defineBatchJob’, users can select a GPU instance. However, if a user wants a specific type, such as a P4, that must be defined here. The default is ‘optimal’.

  • IAM_role (str, optional) – What is the IAM role that the compute instances will assume. The default is None.

Return type:

None.

sendBatchJob(jobName, jobDefinitionName, override_command=None)
Return type:

None

Parameters:

  • jobName (str) – The name for this specific job submission.

  • jobDefinitionName (str) – The name of the job definition to use.

  • override_command (list, optional) – When running this job, what command line argument should be used instead of the command provided in the job definition. The default is None.

Returns:

print(response)
{
'jobId': '876da822-4198-45f2-a252-6cea32512ea8',
'jobName': 'example',
'ResponseMetadata': {
    '...': '...',
    },
}

removeJobDefinition(jobDefinitionName)

This will delete all versions of the job definition.

Return type:

None

Parameters:

jobDefinitionName (str) –

Return type:

None.

defineBatchJob(image_name_uri, host_path, job_command, cpu, memory, jobDefinitionName, gpu=0, timeout=0, environ_vars=None)

Create an AWS Batch job definition that can be utilized by any number of jobs.

Return type:

None

Parameters:

  • image_name_uri (str) – The container end point reference such as ECR location.

  • host_path (str) – If using EKS, host path must be defined.

  • job_command (list) – Default command line arguments, can be overridden for each job submission.

  • cpu (int) – How many vcpu are required.

  • memory (int) – How much RAM in MB is required.

  • jobDefinitionName (str) – Name the definition

  • gpu (int, optional) – How many GPUs are needed. The default is 0.

  • timeout (int, optional) – Job will be auto-aborted after this amount of time (sec). The default is 0.

  • environ_vars (dict, optional) – What environmental variables should be set prior to job command execution. The default is None.

Return type:

None

generateBatch(**kwargs)

Generate the AWS Batch environment and the queue needed to run jobs in AWS Batch.

Parameters:

  • and (Most setup should be set during object instantiation) –

  • batch. (thus a user only needs to call this method to generate the) –

Return type:

None.

terminateBatch()

When a user is done using an AWS Batch environment, this function will terminate both the queue and the batch compute environment.

Return type:

None

Return type:

None

wait_for_jobs_finish(all_jobs)

This is a blocking function that will pause execution of python code until all jobs have terminated within the specific queue.

Return type:

None

Parameters:

all_jobs (list[str]) – Job IDs to monitor for completion. Can be used with the ‘get_running_jobs’ to obtain a list of all currently running jobs in the queue.

Return type:

None

kill_jobs(jobs)

Terminate all jobs provided in this list from a specific queue.

Return type:

None

Parameters:

jobs (list[str]) – List of job IDs that will be terminated. Note, this is often not instances, i.e. the termination command will be sent to Batch, and there may be a delay in seeing termination in the AWS Console as Batch graceful to terminates the container.

Return type:

None

get_running_jobs()

Returns a list that containers both the job name and the batch defined job ID. The ‘kill_jobs’ and ‘wait_for_jobs_finish’ only require the jobID.

Return type:

list

Returns:

[job[‘jobName’] , job[‘jobId’] ]

Return type:

list