14 Feb 2014

How I want to write node: stream all the things!

I wrote the async library back when Node first removed Promises from core (yes, that really was the case). Back then, I preferred to do with plain callbacks what was usually done by including third-party Promise, Future or Continuable libraries instead.

I find a certain elegance in describing complex patterns using simple parts, particularly when the parts are made composable through a common interface. In Node, and JavaScript in general, you're likely to use a combination of Promises, callbacks, Streams, Event Emitters and even ES6 Generators. To me, however, these all represent values in the future. What if we could reduce entire programs to transformations over a Stream, and have just one API to rule them all? Well, here is my proposal, for your consideration and feedback: Highland, a high-level streams library.

var _ = require('highland');

// Mapping over a stream
var doubled = _([1, 2, 3, 4]).map(function (x) {
    return x * 2;

// Reading files in parallel (4 at once)
var data = filenames.map(readFile).parallel(4);

// Handling errors
data.errors(function (err, rethrow) {
    // handle or rethrow error

// Piping to a Node stream

// Pipe in data from a Node stream
var output = fs.createWriteStream('output');
var docs = db.createReadStream();

// wrap a node stream and pipe to file

// or, pipe in a node stream directly:

// Handle events as a stream
var clicks = _('click', btn).map(1);
var counter = clicks.scan(0, _.add);

counter.each(function (n) {

This is not a new idea, but I believe it is a new combination of features, which is important. Those of you that work with Streams in Node may be aware of the excellent modules by Dominic Tarr, similarly if you're from the browser you might use FRP libraries such as RxJS. These are all great, but they hint at a deeper abstraction, one which would allow us to write entire programs using Streams. In an attempt to achieve this, Highland implements:

If you find this idea intriguing and want to explore it further, then check out the Highland website. I'd love to hear your experiences. Source code on GitHub.