Introducing CrunchPipe

Jonathan E. Magen

CrunchPipe is a library for writing modular, computation-focused, optionally-concurrent/parallel programs using the processing pipeline pattern. Borrowing concepts pioneered in the often-neglected fields of dataflow programming and process networks wherein computation can be modeled as a directed graph. CrunchPipe provides support for writing programs in this style using componenets such as streams and pipelines to encapsulate computational steps in a modular fashion.

HTML5 Powered with CSS3 / Styling, Graphics, 3D & Effects, and Semantics



The Problem

Current options

Some mix of the following


Based loosely on 1966 research into the dataflow programming paradigm.

If you can draw your process as a directed graph, you can model your computation!

Essential components


basic pipeline
          pipeline = => true)

          pipeline.bind {|x| x + 1 }
          pipeline.bind {|x| x / 2 }
          pipeline.bind {|x| x * x }

Encapsulation of concurrency/parallelism

parallel pipeline

Keep all state block-local for great justice


basic stream setup

Complete example:

complete basic example
          provider =

          input_stream =

          pipeline = => true)

          pipeline.bind do |element|
            puts "--- Processing #{element}..."
            element + 1

          output_stream =

          end_point = do |data|
            puts "+++ End point got #{data}"

          provider | input_stream
          pipeline < input_stream
          pipeline | output_stream
          output_stream > end_point


Example topology

Let's pre-process data and then transform it while saving the intermediate step's results

example topology

Another valid topology

example giant topology

Streams are observable by anything

Planned improvements

In no particular order

gem install crunchpipe --pre

Contribute at the CrunchPipe GitHub page (it needs work)

References and Further Reading

An Expressive Language and Efficient Execution System for Software Agents
arXiv:1109.2048 [cs.AI]
The semantics of a simple language for parallel programming
Original 1974 Paper
A Non-Standard Semantics for Kahn Networks in Continuous Time
arXiv:1108.5062 [cs.PL]
Requirements on the Execution of Kahn Process Networks
2003 Paper
Dataflow Process Networks
UCB Paper
How to turn a scripting language into a domain specific language for computer algebra
arXiv:0811.1061 [cs.SC]
Enhancing Software Reuse through Application-level Component Approach
Journal of Software [2011, Vol. 6: Issue 3]