大数据的局限性

    Stop me if you've heard this one: Three statisticians go rabbit hunting. They spot a rabbit. The first statistician shoots. He misses the rabbit's head by a foot. The second statistician fires; misses the rabbit's tail by a foot. The third statistician cries out, "We got him!"

    Even if you don't find this joke remotely amusing, you've probably worked with exactly the kind of managerial rabbit hunters it describes. Their math may be impeccable but their real-world results, alas, are rubbish. Lies, damned lies, etc. What must organizations know to improve the odds that their quants will deliver real value instead of statistical illusions? How can stochastically innumerate executives be sure they're not being bamboozled by Big Data?

    Excellent answers can be found in Samuel Arbesman's The Half-Life of Facts and Nate Silver'sThe Signal and The Noise, two distinct but complementary efforts that explore how "data" become "evidence" and why so many sophisticated mathematical models fail so spectacularly at distinguishing the two. The books embrace and extend upon the themes of uncertainty and quantitative self-deception articulated by Nassim Taleb's popular and insightful Fooled By Randomness and The Black Swan, as well as Nobel laureate Daniel Kahneman's superiorThinking, Fast and Slow. Like their precursors, Arbesman and Silver have produced entertainingly actionable books.

    Both authors cite the cynically apt line -- variously attributed to Mark Twain, Will Rogers and Charles Kettering -- that 'It ain't so much the things we don't know that get us into trouble. It's the things we know that just ain't so." Both discuss the media and mechanisms used to distinguish between "real" knowledge and "ain't so's. Arbesman and Silver both argue persuasively that the "ain't so's" are winning. The more data you deal with, the more attention that case deserves.

    Arbesman, an applied mathematician and fellow at Harvard's Institute for Quantitative Social Science, deconstructs what it means to be a fact. He mercifully avoids getting bogged down in post-modernist philosophy. Instead he explores how serious scientists attempt to nail down what it is they think they know about what they're studying. This "scientometric" approach -- the science of how science measures its process and progress -- proves extraordinarily helpful in identifying the lifecycles and ecosystems of what scientists call "facts." This approach allows Arbesman to ask intriguing questions, such as: How are "facts" born? How do they typically replicate, mutate and evolve? How long do they take to die?

    The provocative core of Arbesman's argument is that there is a virtual physics of facts. Depending upon how they're defined and measured, 'facts' follow defined laws and trajectories. "Every day that we read the news we have the possibility of being confronted with a fact about our world that is wildly different from what we thought we knew," he writes. "…But it turns out that these rapid changes, while true phase transitions in our knowledge, are not unexpected or random. We understand how they behave in the aggregate, through the use of probability, but we can also predict these changes by searching for the slower, regular changes in our knowledge that underlie them. Fast changes in facts, just like everything else we've seen, have an order to them. One that is measureable and predictable."