Overview
========

A growing number of applications rely on asynchronous and distributed
processing.

Here are some of the main reasons for considering Amazon SWF and botoflow framework:

* Scalability of the application is the primary motivation for using
  this approach.

  *By designing autonomous distributed components, developers get the
  flexibility to deploy and scale out parts of the application
  independently as load increases.*

* Consumption of cloud services.

  *As application developers take advantage of cloud computing, they
  need to bridge their existing on-premise assets with new assets in
  the cloud.*

* Inherently distributed nature of the process being modeled by the
  application.

  *For example, automating an order fulfillment business process may
  span several systems and human tasks.*

The asynchronous and distributed model has the *benefits of loose coupling and
selective scalability*, but it also *creates new challenges*. Application
developers must coordinate multiple distributed components to get the desired
results. They must deal with the increased latency and unreliability inherent
in remote communication. Components may take extended periods of time to
complete tasks, requests may fail, and errors originating from remote systems
must be handled. To accomplish this today, developers are forced to build
complicated infrastructure that typically involves message queues and databases
along with complex logic to synchronize them. All this "plumbing" is extraneous
to business logic and makes the application code unnecessarily complicated and
hard to maintain.

*Botoflow provides a solution* that makes application development
simple while enabling developers to get all the benefits of asynchronous and
distributed processing in their applications. It accomplishes this through a
powerful programming model layered on `Amazon SWF`_.


Hello World!
------------

Full example can be found here: `examples/helloworld <https://github.com/boto/botoflow/tree/master/examples/helloworld>`_

The following example shows a distributed program written using
botoflow framework. This program *asynchronously calls a remote method*,
which returns a greeting message that is printed to the console.

.. code-block:: python

    @activities(schedule_to_start_timeout=1*MINUTES,
                start_to_close_timeout=1*MINUTES)
    class HelloWorldActivities(object):

        @activity('1.0')
        def get_name(self):
            return input()

        @activity('1.0')
        def print_greeting(self, name):
            print("Hello {}!".format(name))


    class HelloWorldWorkflow(WorkflowDefinition):

        @execute(version='1.0', execution_start_to_close_timeout=1*MINUTES)
        def hello_world(self):
            name = yield HelloWorldActivities.get_name()
            yield HelloWorldActivities.print_greeting(name)

The example above defines a workflow and two activities. Workflows and
activities are the basic building blocks in botoflow:

* Activity is a unit of functionality that is invoked asynchronously
  through `Amazon SWF`_; in other words, it is the *implementation of a
  task*.

* Workflow, on the other hand, is the *control flow* that coordinates
  the execution of activities. Using a workflow you can arbitrarily
  compose tasks, simply by calling activity methods.

For clean separation, the framework uses different base classes/decorators to
define the signature or contract of activities and workflows.

* Activities class is decorated with :py:func:`@activities <botoflow.decorators.activities>`
  and :py:func:`@activity <botoflow.decorators.activity>`.

* Workflow is a subclass of :py:class:`~botoflow.workflow_definition.WorkflowDefinition`.
  The workflow class in our example is ``HelloWorldWorkflow`` and has only one method:
  ``hello_world`` that is decorated with :py:func:`@execute <botoflow.decorators.execute>`.

* Activities, ``get_name`` and ``print_greeting``, are represented by
  methods in the activities class. Workflow implementation invokes
  the ``get_name`` activity to get the user's name/message and calls
  the ``print_greeting`` activity to print a greeting to the console.

It is important to understand that this trivial program is in fact a
distributed application. The workflow and activity implementations can be
hosted on separate machines and scaled independently. In order to call an
activity, *you don't need to instantiate the activities object*, and instead call
it as you would call a Python :py:func:`classmethod`.

You can start an instance of this workflow using the following code snippet:

.. code-block:: python

    session = botocore.session.get_session()

    with workflow_starter(session, aws_region='us-east-1',
                         domain='domain1', task_list='tasklist1'):
        HelloWorldWorkflow.hello_world()  # starts the workflow

Here we use :py:func:`~botocore.session.get_session` from botocore
*(for authentication as well as low-level communication with SWF
service)* and pass it to our
:py:class:`~botoflow.workflow_starter.workflow_starter`.  Then we call
``HelloWorldWorkflow.hello_world()`` in the
:py:class:`~botoflow.workflow_starter.workflow_starter` context to
start a new workflow execution.


.. seqdiag::

    seqdiag flow {
        edge_length = 250;

        Application   -->> WorkflowLogic [label = "HelloWorldWorkflow.hello_world()"];
        WorkflowLogic -->> Activity      [label = "HelloWorldActivities.get_name()"];
        WorkflowLogic <<-- Activity      [label = "return name"];
        WorkflowLogic -->> Activity      [label = "HelloWorldActivities.print_greeting()"]
    }


Non-Blocking Code Using Tasks
-----------------------------

Note that in the previous example, the ``get_name`` activity was returning a ``str``
but when called from within the workflow, it returns
:py:class:`~botoflow.core.future.Future`. When you call this method from within
the workflow, it **returns immediately**. This is because *it only schedules a task*
for execution and **does not block**, waiting for it to complete. This means that
the actual result of the activity is not available to the caller when the call
returns. Therefore, the method returns an object of type
:py:class:`~botoflow.core.future.Future` as a placeholder for the *"future"*
result of executing the remote activity. The actual result is returned only
after the activity has completed, and the time needed to dispatch the task
through `Amazon SWF`_.

One more thing to note in this example is the ``yield`` keyword for both
``get_name()`` and ``print_greeting()`` calls. The ``yield`` keyword in this case is
used to indicate that we want to block and wait for the
:py:class:`~botoflow.core.future.Future` object to complete the call and return
the actual value. You use it to indicate that you need the result of the
activity at that point in the code (in this case to pass the name to
``print_greeting()``, we need the result of ``get_name()`` immediately).


Botoflow and Amazon Simple Workflow Service
-------------------------------------------

Botoflow uses the Amazon Simple Workflow Service (SWF) to schedule
tasks for execution by remote components, to get their results back, and to
store the overall execution state of the application. `Amazon SWF`_ makes it
possible for your application components to be deployed on separate machines
and scaled independently. This also makes the application highly fault tolerant
as it can be executed by multiple processes (workers) and is guaranteed to make
progress if any of them is running.

When using `Amazon SWF`_ directly, you implement the processing steps of your
application as activity workers and the orchestration logic in a decider (also
called a workflow worker). The workers and the deciders also implement code to
poll Amazon SWF and call APIs to provide results and decisions. You then start
workflow executions by calling `Amazon SWF APIs`_ from your application.

A program written using botoflow consists of three types of components:

* Implementation of individual tasks (that is, activities).
* The coordination logic that orchestrates these tasks (the
  :py:class:`~botoflow.workflow_definition.WorkflowDefinition`).
* A component that initiates the coordination logic.

Each of these components can be hosted on separate workers or worker pools and
interact with each other through `Amazon SWF`_. Hence, using the framework, you
can easily create activity workers that host and execute activities, and
workflow workers that host and execute the orchestration logic.


Durable Execution State
-----------------------

In the Hello World example, the activity takes only a few seconds to execute,
but the botoflow allows activities to take arbitrarily long to
complete. For example, an activity may be used to perform complex computation
that takes several hours. In order to reliably execute such long running
processes, the execution state of workflow must be stored durably. The
framework relies on Amazon SWF for this purpose. Amazon SWF maintains the
history (or the state of execution) of each workflow instance. At any point in
time, the history of a workflow instance is a complete and authoritative record
of all the activities scheduled so far, their progress, and results. The
framework uses this history to seamlessly keep track of the progress of the
workflow. This frees you from having to manage the execution state explicitly
in a durable store yourself.

Amazon SWF also provides a reliable communication mechanism between the
workflow and activities. The framework uses it to dispatch tasks to remote
activities and to receive their results. Tasks scheduled in Amazon SWF are
stored durably and are guaranteed to be delivered at most once. When a task
completes, successfully or with error, its results are also stored durably by
Amazon SWF. The framework retrieves these results from Amazon SWF and based on
them proceeds with the execution of your workflow. Amazon SWF guarantees that
the remote activity will either complete successfully or the calling code will
be notified of the failure to complete.

You can also configure the framework to retry a failed task
automatically. These semantics eliminate the need for you to use complex
message passing and queues in your code. You can simply rely on the framework
and Amazon SWF to schedule remote tasks and let them handle the details of
dispatching tasks, retrying failed tasks, and durably storing the results of
their execution. Once a task is complete, the framework receives the results on
your behalf. Since Amazon SWF maintains a durable record of all tasks and their
results, the remote task and its results are not lost if the application
crashes or gets disconnected. Even if all activity workers and deciders go
down, because the execution state is stored by Amazon SWF, the workflow
execution can continue as soon as the activity workers and deciders come back
up.


Distributed Execution
---------------------

In essence, each workflow instance is a virtual thread of execution. This
virtual thread of execution may span activities and orchestration logic running
on several remote machines. Amazon SWF and the framework act as the operating
system that manages these threads on a virtual CPU. It keeps the state of
execution of the thread, switches between threads, and knows how to resurrect a
thread back to the point at which it was switched out. As remote activities
complete, the framework looks at the history and replays the workflow logic,
plugging in the results of completed tasks. As tasks complete, the workflow
logic makes more progress each time it is replayed. Since the workflow logic
invokes activities, which may be remote and long running, the framework does
not replay them. Instead, it plugs in the results that activities returned
using the history.

This ability to resurrect the program from state stored in Amazon SWF means
that the program is stateless and you can run it on many machines for the
purpose of scalability. The program can be initiated independently of workers'
availability because the initiation is managed by Amazon SWF. The program is
highly scalable as any number of instances can be created in parallel. Requests
to execute activities are delivered to workers through dynamically allocated
consistent logical queues called task lists; therefore, the work is
automatically load balanced among worker processes. Amazon SWF uses the HTTP
long poll mechanism to deliver tasks to workers allowing them to pull tasks at
their own pace. This ensures that workers are not overloaded even if there is
an unexpected spike in requests. The HTTP poll mechanism also allows your
workers to run behind firewalls since you are not required to open externally
visible ports. This allows your applications to use resources in the cloud as
well as on on-premise data centers.

Together, botoflow and `Amazon SWF`_ make it easy to create scalable
and fault tolerant applications that perform asynchronous tasks that may be
long running, remote, or both.

.. _Amazon SWF:  https://aws.amazon.com/swf/
.. _Amazon SWF APIs: http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dg-using-swf-api.html