OFlow is the latest in a series of process flow tools I've worked on over the years going back to my first one at TIBCO Software. OFlow relies on queues, immutable data, and a separate thread for each task in a process flow. It is build to allow highly parallel processing.

GemChart is a somewhat non trivial example of how OFlow can be used. It also gave me an excuse to put together a graphing application for charting downloads of my gems. It was fun to put together and gave me a chance to test out the debugging and tracing tools in OFlow.

What is OFlow?

A process flow or just plain "Flow" for short is a way of describing a set of related processing steps much like an application or program. A Flow is composed of separate steps of tasks that are evaluated according to the layout of the Flow and linkages between the tasks. Data is passed between tasks until the flow through the tasks completes. Just like any other software program.

Tasks are similar to functions in many languages except they do not return but instead propogate data to the next Task. There is no going back, only forward. To move forward a Task sends data along a link to another Task. The data in OFlow is encapsulated in an immutable Box. Tasks a connected by named Links. A Task does not know where it is sending data, a Box to. It just sends it on one of it's named Links. That allows Tasks to be independent and unaware of the overall Flow construction. In many ways a Task is analogous to a person that performs a specific task. Imagine a person in an office that receives packages, processes the contents of the package and then sends out new packages to others.

Links can be thought of as pipes from one office to another or one Task to another Task. They are attached on the sender side by a named end point and terminate at another office or Task. The Link can also place a label or operation on the Box being sent over a Link.

Data send between Tasks are encapsulated in an immutable Box. The Box and contents can not be changed so the sender is protected from having it's data change by some other Task. Boxes do support making copies of content that is to be changed while not duplicating data that is not changed. Boxes can also be tracked. The ability to track a box aids in debugging and monitoring the processing of the system.

Processing start by the generation of event. Events are generated by a Task that has some connection to an external element such as a clock or socket. In the GemChart example there are two entry point to the flow. One is a timer and the other is a simple HTTP server.

GemChart Application

The GemChart application gathers download statistics from the RubyGems web site and stores them in a very simple data store. It also responds to HTTP requests to chart or graph the download results over time. It's a fairly simple application but it does make use of several features of OFlow.

Drawing a Flow

A Flow in OFlow can be created with Ruby code or more elegantly and simply by drawing the Flow as a diagram. As of the current release, 0.8.0 only the OmniGraffle format is supported. OmniGraffle has the advantage of being able to save files in XML format. OFlow can read these files and use them to execute Flows directly.

The GemChart Flow has two triggers. One is a timer that kicks off every 86400 seconds starting at 23:11:00 which means the trigger fires once a day at 11:11 PM. This Scheduler Task sends a Box to the GetStatus Task which uses the RubyGem REST APIs to gather download data about the identified gems.

The text in a Task shape have special meaning to the OFlow loader. The first line is the name of the Task. Following that are attributes of the Task. A reserved attribute key is "class". The class attribute identified the Ruby class to use as the implementation of that Task. The other attributes are passed to that implemenation on initialization.

Proceeding with the GemChart flow triggered by the Scheduler the Link named as "save" is used to deliver an package with an operation of "insert_update" to the Store Task. The Store Task is an instance of a build in OFlow class named Persister. The Persister class provides a very simple file based storage capability. The data collected from RubyGems is saved as files by the Store task. Each gem download record is stored as a separate JSON formated file.

The second trigger Task is also an OFlow buildin Task, the HttpServer Task. It listens on a port and passes the HTTP request, a GET in this case to the "get" Link. This GET package is received by the Switch Task that decides if the request is a request for a static page or for JSON data.

Static page requests go to the StaticPager Task which then sends the response back to the HttpServer Task so it can respond to the original GET request.

Requests for JSON data go to the Store Task with a "query" operation. The Store Task then populates a data package that is sent to the PageMaker Task.

The PageMaker Task reformats the query result data into a set data points expected by the Javascript on the we page served by the StaticPager. The Javascript then draws the graph depicted in the GemChart Graph diagram.

Code the Tasks

There are a few custom Tasks in the GemChart application. The GemStatus, Switch, StaticPager, and PageMaker Tasks are custom Tasks. Each is written in Ruby and can be treated as a standalone component that can be tested with the classes in the ::Oflo::Test module.

Input to the Tasks is through the perform(op, box) method. Output is performed by calling ship() on the task attribute of the Task actor. The code is provided in the OFlow example entitled gemchart in the oflow-ruby repo on github

Further debugging and monitoring is possible during integration with a -v option. Increasing the verbosity shows more about what tasks are executing and what is passing across the links.

Web Pages

I'm not very adept at Javascript so the Javascript used for the GemChart is probably not of the highest caliber. It does the job well enough to display the graph for the example though. The page is set up to display any of the six gems chosen to be part of the example. The approach was to have the Javascript on a static page request the data points as set of JSON arrays that were then used to draw a simple graph.

No third party packages were used as the example is meant to be standalone. To make page resizes faster data is stored in a hidden input element.

Buttons change the target gem and cause a redraw with the new fetched data. Thats about all there is to it.

Summary

Creating the example was helpful in refining some of the debugging tools and in evaluating how easy or complex is was to design a highly parallel application. I've been using developing with process flows for a while so it came pretty natural to me. For others it might not be as easy to visualize how separate tasks can be made to work together when each task is operating independent of others. OFlow attempts to make the experience easier and more fool proof. In the end it was fun and rewarding to put together an application that I now keep running to track the download history on my gems on RubyGems.