Many years ago I researched explosives by shining a light on them. It was every bit as exciting as it sounds. We would shine a light, take a picture, then study the explosive to see if it changed. I would painstakingly scour thousands of data points, looking for small fluctuations in intensity, signs of discoloration, or any statistically significant feature. We collected immense amounts of data from sensors, but the explosive always looked the same when we took snapshots. Then eventually we found out that if we looked not just at the snapshots, but also at the differences between the snapshots using a mathematical formula, we could see dramatic changes. We found out that every explosive was different, and we could effectively detect an explosive from a distance by just shining a light. Today, that research is being used to scan people before they enter airports for bombs.

Today, companies have more customer data than they can handle. Like a digital version of the show Hoarders, companies try to keep every bit of detail for as long as possible with the hope that one day these useless bits can be turned into massive new revenue opportunities. Over the past five years, bright engineers have devised open-sourced solutions to store and process the data deluge. We now even have a “big data stack” — that is, a framework for commoditizing data. Continue Reading…