Your data isn't big

Adam Drake’s Command-line tools can be 235x faster than your Hadoop cluster is from last January, but it made the Hacker News front page on Sunday and I’ve had it sent to me three times since. If you haven’t read it, the setup is simple. Someone had used Amazon’s Elastic MapReduce to count wins and losses across a pile of chess games, about 1.75GB of PGN files, and it took 26 minutes on 7 machines. Drake did the same job on his laptop with find, xargs and mawk, on about twice the data, in roughly 12 seconds.

The trick in his pipeline is mostly that it never stops moving. Each mawk reads its files once, keeps three counters, and prints one line at the end, and xargs -P4 runs four of those at a time. Nothing gets written to disk or shipped over a network in between. (And nobody has to wait for seven JVMs to start before any work happens.)

I have a confession that goes with this. About eighteen months ago we set up an EMR job to crunch our web logs every night: top endpoints, slow requests, error rates per customer. It felt like the grown-up thing to do, everyone was talking about Hadoop, and our logs were “big”, by which I mean they were bigger than anything I wanted to open in a text editor. The job took about 40 minutes, failed maybe once every couple of weeks for reasons that were never the same twice, and one engineer on the team owned it the way you own a car that needs a lot of work.

Last fall that engineer got fed up and rewrote the main report as this, more or less:

zcat access-*.log.gz | awk '{print $7}' | LC_ALL=C sort | uniq -c | LC_ALL=C sort -rn | head -50

That’s the top 50 request paths for a day (field 7 in the combined log format). It ran in under four minutes on one box. The LC_ALL=C matters more than you’d think, by the way, because it makes sort compare bytes instead of doing locale-aware collation, and on our data that was most of the difference. Once we had a few of these, the EMR job quietly got turned off.

The number that surprised me when I actually worked it out is how much data actually fits on one machine now. A decent laptop SSD reads at something like 500MB/s, so you can stream 100GB through a pipeline in a bit over three minutes if the CPU keeps up. Our entire year of compressed logs fits on one drive. You can buy a server with a quarter of a terabyte of RAM. “Big data” should mean the stuff that doesn’t fit, and for most of the startups I talk to, it fits.

I’m not anti-Hadoop. If you really have petabytes, or a hundred teams sharing a cluster, it earns its complexity. And Drake’s job splits cleanly by file with no coordination at all, which is the easy case. Joins across huge tables are a different problem and I wouldn’t try those with awk. But I’d make people show me the data size before they get to say “big”.

Growing up on DOS with 640K, you learned early to stream things through instead of loading them, because there was no other way. It’s funny that the same habit is now the fast option.*

* Drake also points out that mawk was much faster than gawk for his job. I tried it on our logs and got a smaller but real speedup. Worth checking which awk you actually have.