Thursday, June 26, 2014

The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google : Paper Trail

More on the MR-RIP theme (via Mike Olson)

"It was known for decades that generalised dataflow engines adequately capture the map-reduce model as a fairly trivial special case. However, there was real doubt over whether such engines could be efficiently implemented on large-scale cluster computers. But ever since Dryad, in 2007 (at least), it was clear to me that Map-Reduce’s days were numbered. Indeed, it’s a bit of a surprise to me that it lasted this long.

Map-Reduce has served a great purpose, though: many, many companies, research labs and individuals are successfully bringing Map-Reduce to bear on problems to which it is suited: brute-force processing with an optional aggregation. But more important in the longer term, to my mind, is the way that Map-Reduce provided the justification for re-evaluating the ways in which large-scale data processing platforms are built (and purchased!)."
The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google : Paper Trail

No comments: