big data with node.js, sqlite, redis, polymaps
big-ish data with node.js, sqlite, redis, polymaps, and maps
by me, @tmcw
problem: openstreetmap is not quite complete
problem: openstreetmap is not quite complete
and people notice when it's their town that's missing
so: take our global base layer, find its density at a 3mi resolution
aka a hacky list-indexable hash table
'MapBox Streets'
~240 gigabyte
SQLite database of tile images
→ 4.6MB SQLite database of street density
Flow-control stuff breaks usually
you can't keep track of all callback arg-results
function run(s) {
  asyncFn(function(err, rows) {
    // do stuff
    if (step < stepend) {
      step++; run(step);
} run(step = 0);
crossing the bindings gap
for (var i = 0;
  i < 1e9; i++) {
  var x = callToCLib(i, {
    start: i,
    limit: 1
  // do stuff with x
for (var i = 0;
  i < 1e9; i += 100) {
  var x = callToCLib(i, {
    start: i,
    limit: 100
  // do stuff with x
(aka loop unrolling, duff's device, batching)
redis is a darn good hammer
for this project: needed a big hash of incremented numbers
rclient.incrby(key, val);
rclient.sadd('keys', key);
  function(err, k) {
    rclient.get(k, function() {
redis ♥ numbers and is super-fast for prototyping
next: prototyping a visualization
next: prototyping a visualization (aka doing it live)
tactic: use canvas
tactic: use canvas
abstract only when necessary
tactic: use canvas
with node-canvas as your escape hatch to other sorts of output
fast enough, pixel-perfect, easy-ish to use
polymaps stays out of your way.
// this is a valid layer
po.layer(function(tile) { }))
iterate in-browser: colors & scales are finicky
iteration speed > screen speed
when you're done: baking.
porting stuff to node: only stickler is d3
but it's so worth it
var scale = d3.scale.linear()
  .range([0, 1]).clamp(true)
  .domain([0, 10000]);
follow the dependencies, and
var d3 = require('./d3_compat.js')
bake tiles: using node-sqlite3 for dbing, node-canvas for rastering
a few hours later