Introducing CrunchPipe

Jonathan E. Magen

• Sunday, October 16th, 2011
• NewHaven.rb

CrunchPipe is a library for writing modular, computation-focused, optionally-concurrent/parallel programs using the processing pipeline pattern. Borrowing concepts pioneered in the often-neglected fields of dataflow programming and process networks wherein computation can be modeled as a directed graph. CrunchPipe provides support for writing programs in this style using componenets such as streams and pipelines to encapsulate computational steps in a modular fashion.

Who

• Director of Core Engineering @ Continuity Control
• Did research at The Evergreen State College Advanced OS Lab
• Likes to cook
• Super-huggable

The Problem

• Ruby has a bad rep when it comes to computationally-intensive applications
• Ruby lacks adequate library support for modeling complex computational processing in accordance with accepted patterns
• Code organization and separation of concerns can be difficult in the absence of well-defined practices and patterns
• Concurrency is hard, yo

CrunchPipe

Based loosely on 1966 research into the dataflow programming paradigm.

If you can draw your process as a directed graph, you can model your computation!

• Observer pattern for easy integration with other code
• Concurrency/parallelism (with the parallel gem) encapsulation
• Intuitive (hopefully) DSL for coordination

Pipelines

• Bind functions to pipelines
• Binding order preserved

          pipeline = CrunchPipe::Pipeline.new(:parallel => true)

          pipeline.bind {|x| x + 1 }
          pipeline.bind {|x| x / 2 }
          pipeline.bind {|x| x * x }
        

Encapsulation of concurrency/parallelism

• Concurrent/parallel processing happens automatically, inside the pipeline
• Order of operations is always preserved
• Data is sent through the pipeline in parallel and reintegrated before being sent onward

Keep all state block-local for great justice

Streams

• Feed input to and recieve data from pipeline(s)
• Observer pattern
• Streams are observable by anything

Complete example:

          provider = CrunchPipe::DataProvider.new

          input_stream = CrunchPipe::Stream.new

          pipeline = CrunchPipe::Pipeline.new(:parallel => true)

          pipeline.bind do |element|
            puts "--- Processing #{element}..."
            element + 1
          end

          output_stream = CrunchPipe::Stream.new

          end_point = CrunchPipe::DataEndPoint.new do |data|
            puts "+++ End point got #{data}"
          end

          provider | input_stream
          pipeline < input_stream
          pipeline | output_stream
          output_stream > end_point

          provider.provide([1,2,3,4,5])
        

Example topology

Let's pre-process data and then transform it while saving the intermediate step's results

Another valid topology

Streams are observable by anything

Planned improvements

In no particular order

• Bake in distributed features
  • Possibly using ØMQ
  • Kahn Process Networks
  • Petri Nets
• Improved DSL
• RSpec matchers and expectations to aid testing
• Benchmark suite
• Pluggable components for extra functionality
  • Memoization
  • Configurable logging
  • Statistical analysis/reporting

gem install crunchpipe --pre

Contribute at the CrunchPipe GitHub page (it needs work)

References and Further Reading

An Expressive Language and Efficient Execution System for Software Agents

arXiv:1109.2048 [cs.AI]

The semantics of a simple language for parallel programming

Original 1974 Paper

A Non-Standard Semantics for Kahn Networks in Continuous Time

arXiv:1108.5062 [cs.PL]

Requirements on the Execution of Kahn Process Networks

2003 Paper

Dataflow Process Networks

UCB Paper

How to turn a scripting language into a domain specific language for computer algebra

arXiv:0811.1061 [cs.SC]

Enhancing Software Reuse through Application-level Component Approach

Journal of Software [2011, Vol. 6: Issue 3]

Credits

• Slides made with deck.js
• Diagrams (also the goofy ones) made with GraphViz
• Syntax highlighting by SHJS
• Brought to you by the number 42 and the letter Q
• HTML5 markup by the W3C and WHATWG
• JavaScript by jQuery
• Soup of the day is cream of broccoli
• Deployed with capistrano

Fin