Spinning Your Workflows With Cadence

1. What Is Cadence?

(Source: Shutterstock)

Pedaling with a high Cadence (pedal revolutions) is called Spinning; Mashing or Grinding is slow (and bad). 

Over the last decade, the rapid rise in Big Data use cases has accelerated popular massively scalable open sources technologies such as Apache Cassandra® for storage, Apache Kafka® for streaming, and OpenSearch® for search. In addition, there’s a new member of the peloton, Cadence, for workflow orchestration.

My flatmate used to carry his Cello on a bike to music lessons, a bit like this, but music and cycling also have something more in common—Cadence!

(Source: Adobe Stock)

In music, a Cadence is a musical phrase that sounds complete.  And in cycling, Cadence is how fast the cyclist pedals, which influences efficiency and speed.

Workflow orchestration is the process of defining and executing a series of tasks and the order they are to be carried out. For example, you may want some tasks to be performed in sequence, but others may be performed concurrently. You also see this idea in orchestral music scores, such as in this example, which provides the conductor, musicians, and choir instructions on what notes to play and when.

By Max Reger—IMSLP, Public Domain, (https://commons.wikimedia.org/w/index.php?curid=51020706)

In typical workflow terminology (e.g., BPMN), a workflow has start and end events, activities (atomic or compound tasks), gateways (decisions), and sequences or flows—the order of activities in a workflow.

In bygone eras of computing, I’ve experienced previous attempts to implement and model workflows, including Enterprise Java Beans (Stateful Session Beans were used for workflows), ebXML, BPEL, and modeling evaluating workflows using discrete event simulation. Traditional workflow modeling uses the semantics of semi/formal notations, including Finite State Machines (FSM), Markov Models, discrete event simulation, and Petri networks to specify, visualize, and analyze workflows.

Open source Cadence was developed by Uber and based on the AWS SWF (Simple Workflow Service). It uses Apache Cassandra (and other databases) for persistence and is designed to be highly fault-tolerant and scalable. Perhaps the most surprising feature of Cadence, compared to different workflow engine approaches I’ve encountered previously, is that it is focused on helping developers write workflows primarily either in Java or Go (other languages are available) and therefore doesn’t come with any visualization notation or tool to specify workflows. Instead, the semantics are just plain old code. However, it does have the ability (with the Cadence Web client) to visualize workflows as they execute. 

(Source: Shutterstock)

Now, riding a bike needs a lot of gear. For example, you need a bike (logically), helmet, shoes, water bottle, glasses, gloves, a light, pump, lycra (optional!), etc. Likewise, successful use of Cadence needs many things, including workflows, activities, domains,  workflow clients, and workers. Let’s take a look at each in turn.

2. Cadence Workflows

Factory workflows require a strict ordering of tasks

(Source: Shutterstock)

Let’s start with the most basic Cadence concept, workflows. I’ll be using the cadence java client, which requires you to download it and configure it to compile in your IDE of choice (I use Eclipse and Maven, and for simplicity, I’ve omitted the imports). There are a bunch of java client examples that inspired mine. But, first, we need a workflow interface and implementation.

Workflow interface:

static final String activityName = "ExampleActivity";



    public interface ExampleWorkflow {

        @WorkflowMethod(executionStartToCloseTimeoutSeconds = 120, taskList = activityName)

        void startWorkflow(String name);

    }

Workflow implementation:

public static class ExampleWorkflowImpl implements ExampleWorkflow {

        

        private ExampleActivities activities = null;

        

        public ExampleWorkflowImpl() {

            this.activities = Workflow.newActivityStub(ExampleActivities.class);

        }

        

        @Override

        public void startWorkflow(String name) {

            System.out.println("Started workflow! ID=" + Workflow.getWorkflowInfo().getWorkflowId());

            

            String r1 = activities.task1(name);

           String r2 = activities.task2(r1);

            

            System.out.println("Workflow result = " + r2 + " ID=" + Workflow.getWorkflowInfo().getWorkflowId());

        }

       

    }

There’s always precisely one method in Cadence with the @WorkflowMethod annotation, which acts as the start event for the workflow and contains the sequence flow and activities. Calling it starts a stateful workflow instance, which eventually ends.

In the implementation, the method startWorkflow() calls two other methods in sequence (task1, task2), giving us the workflow logic  (simple sequential) and 2 activities. 

Each running workflow instance has a unique ID, which we’ve used in the above example. You need a workflow ID as workflows can have multiple instances. 

However, there is some more complexity here. In reality, Cadence isn’t just a POJI/POJO; it is a platform to execute the stateful workflows scalably and reliably. 

You’ll notice a timeout and activities and taskList above. Timeouts are needed as workflows, like music and cycling, must eventually end.

3. Cadence Activities

Some people perform strange activities on bikes—this one is likely to fail!

(Source: Shutterstock)

Activities are a core feature of Cadence. Workflows are stateful and fault-tolerant, but you will eventually want to call remote APIs to perform tasks in a distributed microservices architecture. However, remote API calls can fail and can’t be made fault-tolerant as they are external to the Cadence workflow. The solution Cadence uses to allow any code, including remote calls, to be wrapped in a Cadence Activity, with the caveat that Cadence doesn’t recover the activity state on failure. Activities are guaranteed to be executed at once (idempotent activities will be automatically retried on failure, non-idempotent activity failures will need to be handled by business-specific logic, including compensation activities, more information on RetryOptions can be found here).

Activities are just a POJO/POJI pair containing the tasks/methods to be executed. Each method has a @ActivityMethod annotation with options. 

public interface ExampleActivities

    {

        @ActivityMethod(scheduleToCloseTimeoutSeconds = 60)

         String task1(String name);

        @ActivityMethod(scheduleToCloseTimeoutSeconds = 60)

         String task2(String name);

    }



public static class ExampleActivitiesImpl implements ExampleActivities

    {

         public String task1(String arg) {

             System.out.println("task1 " + arg + " in " + Activity.getWorkflowExecution().getWorkflowId());

             return arg+"task1";

         }



    public String task2(String arg) {

             System.out.println("task2 " + arg + " in " + Activity.getWorkflowExecution().getWorkflowId());

             return arg+"task2";

         }    

    }

We’ve already registered the Activities in the workflow constructor above. Note that a slightly different way is required to get the WorkflowId in an activity. So is a Workflow and Activities all we need to get it running? 

4. Cadence Domains

Kingdom: a Domain ruled by a Monarch (if you are ever stuck for a crossword clue!)

(Source: Shutterstock)

Cadence has the concept of domains, which is just a namespace that workflows live in. You have to create or reuse an existing domain before you can start workflow or workers. Here’s an example of a method to register a new domain (or return if it exists):

public static void registerDomain(String host, String domain)

    {       

        String nameDescription = "a new domain";

        

         IWorkflowService cadenceService = new WorkflowServiceTChannel(ClientOptions.newBuilder().setHost(host).setPort(7933).build());

        RegisterDomainRequest request = new RegisterDomainRequest();

        request.setDescription(nameDescription);

        request.setEmitMetric(false);

        request.setName(domain);

        int retentionPeriodInDays = 1;

        request.setWorkflowExecutionRetentionPeriodInDays(retentionPeriodInDays);

        try {

          cadenceService.RegisterDomain(request);

          System.out.println(

              "Successfully registered domain "

                  + domain

                  + " with retentionDays="

                  + retentionPeriodInDays);

        } catch (DomainAlreadyExistsError e) {

          System.out.println("Domain " + domain + " is already registered")

    }

And just in case you are wondering what IWorkflowService and WorkflowServiceTChannel are (I did, and we’ll also use these below)! These are in the ServiceClient package, which doesn’t seem well documented, but this appears to be the primary way you connect to the Cadence server.

“The ServiceClient is an RPC service client that connects to the cadence service. It also serves as the building block of the other clients.”

I found the following client types (in the client package and serviceclient package): client/ActivityCompletionClient, client/WorkflowClient, serviceclient/IWorkflowService (There may be more).

5. Cadence Workflow Client

How do you communicate with the Cadence server?

(Source: Shutterstock)

We’re now ready to try the example. In the primary method, here is some sample code to create a domain and a new WorkflowClient instance:

String host  = "Cadence Server IP";

        

String domainName = "example-domain";

        

registerDomain(domainName);

        

WorkflowClient workflowClient =

                WorkflowClient.newInstance(

                        new WorkflowServiceTChannel(ClientOptions.newBuilder().setHost(host).setPort(7933).build()),

                        WorkflowClientOptions.newBuilder().setDomain(domainName).build());

6. Cadence Workers

I didn’t think you could have “too many” bikes until I visited the Netherlands and encountered hordes of cyclists—bells replace helmets as the essential accessory! 

(Source: Adobe Stock)

If you try and run the above example, you’ll find that none of the tasks are progressed, and the workflow times out eventually. What’s gone wrong? The way that activities are executed is by Workers. Workers for the activities need to be created before any work is performed (similar to musicians in an orchestra—you can have a conductor and the score, but the score isn’t played until the musicians are on the stage with their instruments, ready to perform, and the conductor starts them off).

So now that we have a WorkflowClient, we’re ready to create a worker. First, you have to specify the String activityName, and register both the Workflow and Activities implementations with the workflow before starting it (if you forget to register the activities, there’s no error, but nothing happens as Cadence will wait for a worker process to appear, at least until the workflow times out!):

WorkerFactory factory = WorkerFactory.newInstance(workflowClient);

        Worker worker = factory.newWorker(activityName);

        worker.registerWorkflowImplementationTypes(ExampleWorkflowImpl.class);



        worker.registerActivitiesImplementations(new ExampleActivitiesImpl());

        

        factory.start();

Note that the workers would typically be run in a separate process to the workflow and for scalability reasons on well resourced (and multiple) servers. So this is where the actual workflow task “work” is really being done. 

But how many workers do we have? There are default concurrency and rate limits, and the default settings from WorkerOptions are:

• Maximum number of concurrent activities = 100

• Maximum number of concurrent workflows = 50.

• Maximum number of concurrent local activities = 100

• Maximum number of activities started per second = 0 (unlimited)

I suspect that if you were clever and had sufficient monitoring of the workflows and activities (number and times in each), you could use Little’s Law (which relates concurrency, throughput, and time) to estimate the magic numbers more accurately. Here’s an example of changing the WorkerOptions defaults, including an explanation of what makes sense.

7. Cadence Workflow Execution: Synchronous or Asynchronous

(Source: Shutterstock)

It’s now time to start executing a workflow instance as follows:

ExampleWorkflow exampleWorkflow = workflowClient.newWorkflowStub(ExampleWorkflow.class);

exampleWorkflow.startWorkflow("workflow 1");

Workflows can be started synchronously or asynchronously. For example, twostart two instances synchronously:

ExampleWorkflow exampleWorkflow = workflowClient.newWorkflowStub(ExampleWorkflow.class);

exampleWorkflow.startWorkflow("workflow 1 sync");



ExampleWorkflow exampleWorkflow2 = workflowClient.newWorkflowStub(ExampleWorkflow.class);

exampleWorkflow2.startWorkflow("workflow 2 sync");

And asynchronously: 

ExampleWorkflow exampleWorkflow3 = workflowClient.newWorkflowStub(ExampleWorkflow.class);



CompletableFuture<String> r3 = WorkflowClient.execute(exampleWorkflow3::startWorkflow, "workflow async");

CompletableFuture<String> r4 = WorkflowClient.execute(exampleWorkflow3::startWorkflow, "workflow 2 async");



          try {

            r3.get();

            r4.get();

        } catch (InterruptedException e1) {

            e1.printStackTrace();

        } catch (ExecutionException e1) {

            e1.printStackTrace();

        }

Just remember to wait for the asynchronous workflows to complete (as in this example using get()); otherwise, nothing much will happen (as the workers are in the same thread in this example).

8. Cadence Asynchronous Activities and Blocking

(Source: Shutterstock)

Workflows aren’t the only things that can be called asynchronously; activities can also be executed concurrently. This is helpful when you want to run a large number of concurrent tasks at once and then wait for them all to complete. For example, let’s replace the original startWorkflow(String name) method with a new method startConcurrentWorkflow(int max), and we’ll reuse the original task1() activity but execute it concurrently:    

public void startConcurentWorkflow(int concurrency) {

            List<Promise<String>> promises = new ArrayList<>();

            List<String> processed = null;

            try {

                          for (int i=0; i<concurrency; i++)

                {

                               Promise<String> x = Async.function(activities::task1, “subpart “ + i);

                    promises.add(x);

                }

                

                // allOf converts a list of promises to a single promise that contains a list

                // of each promise value.

                Promise<List<String>> promiseList = Promise.allOf(promises);



                // get() blocks on all tasks completing

                List<String> results = promiseList.get();

                

                int count = 0;

                for (String s: results)

                {

                    System.out.println("Processed " + s);

                    count++;

                }

                System.out.println("Processed “ + count + " concurrent tasks");

            } finally {

                }

            }

        }

Note that we have to use the Async.function(activities::activityName, arg) method and the Promise class with methods add(), allOf(), and get() to block correctly until all of the activities are complete. 

9. Cadence Restrictions

I’m guessing this sign bans tractors, horses and carts, and bicycles?!

(Source: Shutterstock)

The way that Cadence manages workflow state is “clever”’! Rather than maintaining the condition of each stateful workflow instance in a database (which is a typical approach for persistence and reliable workflow engines but has scalability limitations), Cadence emits state changes to the database as historical events. Then if something goes wrong (or even if you sleep in a workflow), it just restarts the workflow and replays the history to determine what the next task is to run. This uses the event sourcing pattern (similar to how Kafka Streams can compute state from event streams or event streams from the state).

This approach supports large numbers of workflow instances simultaneously (which is likely for long-running and high throughput workflows). Still, it does have negative implications for long workflows, which require increasingly large histories and replaying of many events to regain the correct state.

However, this approach does introduce some critical restrictions on workflow implementation that you need to be aware of—basically, they have to be deterministic. The Cadence Workflow class has many practical methods that can be used in place of unsafe/non-deterministic/side-effect riddled standard Java methods, particularly for the time, sleeping, and random numbers.

So that’s just about all I know about Cadence so far. However, there’s a lot more to discover, including how it handles exceptions, retries, long-running transactions, different sorts of timeouts, compensation actions, signals, queries, child workflows, and how you can call remote APIs and integrate it with Kafka (e.g., to send and block/receive a message as an activity).