Introducing CrunchPipe

Jonathan E. Magen

CrunchPipe is a library for writing modular, computation-focused, optionally-concurrent/parallel programs using the processing pipeline pattern. Borrowing concepts pioneered in the often-neglected fields of dataflow programming and process networks wherein computation can be modeled as a directed graph. CrunchPipe provides support for writing programs in this style using componenets such as streams and pipelines to encapsulate computational steps in a modular fashion.

HTML5 Powered with CSS3 / Styling, Graphics, 3D & Effects, and Semantics

Who

yonkeltron-avatar

The Problem

Current options

Some mix of the following

CrunchPipe

Based loosely on 1966 research into the dataflow programming paradigm.

If you can draw your process as a directed graph, you can model your computation!

Essential components

Pipelines

basic pipeline
          pipeline = CrunchPipe::Pipeline.new(:parallel => true)

          pipeline.bind {|x| x + 1 }
          pipeline.bind {|x| x / 2 }
          pipeline.bind {|x| x * x }
        

Encapsulation of concurrency/parallelism

parallel pipeline

Keep all state block-local for great justice

Streams

basic stream setup

Complete example:

complete basic example
          provider = CrunchPipe::DataProvider.new

          input_stream = CrunchPipe::Stream.new

          pipeline = CrunchPipe::Pipeline.new(:parallel => true)

          pipeline.bind do |element|
            puts "--- Processing #{element}..."
            element + 1
          end

          output_stream = CrunchPipe::Stream.new

          end_point = CrunchPipe::DataEndPoint.new do |data|
            puts "+++ End point got #{data}"
          end

          provider | input_stream
          pipeline < input_stream
          pipeline | output_stream
          output_stream > end_point

          provider.provide([1,2,3,4,5])
        

Example topology

Let's pre-process data and then transform it while saving the intermediate step's results

example topology

Another valid topology

example giant topology

Streams are observable by anything

Planned improvements

In no particular order

gem install crunchpipe --pre

Contribute at the CrunchPipe GitHub page (it needs work)

References and Further Reading

An Expressive Language and Efficient Execution System for Software Agents
arXiv:1109.2048 [cs.AI]
The semantics of a simple language for parallel programming
Original 1974 Paper
A Non-Standard Semantics for Kahn Networks in Continuous Time
arXiv:1108.5062 [cs.PL]
Requirements on the Execution of Kahn Process Networks
2003 Paper
Dataflow Process Networks
UCB Paper
How to turn a scripting language into a domain specific language for computer algebra
arXiv:0811.1061 [cs.SC]
Enhancing Software Reuse through Application-level Component Approach
Journal of Software [2011, Vol. 6: Issue 3]

Credits

Fin

kthxbai