Mining Big Data

Looking for a business advantage? Take a cue from Moneyball, which dramatized how sports teams could win if they played by the numbers instead of gut instinct. Regardless of the size of your business, it’s time to get on top of the relentlessly growing and invaluable information stream generated by nearly every sector of society. Whatever software you’re using to process data today is almost certainly inadequate to meet the challenge of a world that’s starting to think in zettabytes (that’s 1 billion terabytes, with each terabyte being 1 trillion bytes). The challenge is not just to store all that information, but to understand the opportunities it offers and effectively analyze it ahead of the competition.

Big Data, as it’s come to be called, refers to large data sets that come from just about everywhere—including online sales records, shipping information, climate information, satellite photos and remote surveillance video, computer-generated stock market trades, arrest records, posts to social media sites, flight information, cellphone GPS signals… and much more.

• Police departments routinely sift through huge volumes of such information to predict and plan for crime trends. They may look, for instance, at weather, traffic patterns, sporting event schedules, holidays and dates of paydays to pinpoint crime hot spots where targets of opportunity (like distracted people flush with cash) intersect with would-be bad guys.
• Savvy retailers can evaluate sales performance of products, pricing trends and demographics to better understand their customers’ rapidly shifting needs.
• Lawyers could study individual judges’ decisions to gain insights into strategies to use in their courtrooms—in far less time than it would take them in the analog law library.
• Airlines can know before a plane lands that a passenger’s baggage didn’t make the flight, then alert the passenger about the bag’s whereabouts and when he will get it, before the passenger’s blood begins to boil as he waits next to an empty carousel.
• And athletic team managers can analyze data and stats to identify undervalued players, as in the Moneyball example based on the Oakland A’s baseball team, as chronicled in the 2003 book by Michael Lewis and last year’s movie starring Brad Pitt.

If you’ve never heard of Big Data or its importance, it’s no wonder. Consider that 90 percent of the world’s data was created in the last two years, IBM tells us, with more than 2.5 quintillion bytes of data being created daily.

Just a year ago, jobs that involve crunching Big Data barely existed, but now the United States faces a shortage of up to 190,000 workers with analytical expertise, as well as 1.5 million managers and analysts to understand and make decisions based on that analysis, according to the McKinsey Global Institute, the research arm of the international management consultant McKinsey & Co.

The market for Big Data technology and services will grow from $3.2 billion in 2010 to $16.9 billion in 2015, according to a 2012 report from the forecasting company International Data Corp. The growth is even higher in certain sectors such as storage, estimated by IDC to be 61.4 percent over the next five years. And specialized data handlers will pioneer new markets; companies that provide clinical medical information, for example, could see a market worth more than $10 billion by 2020, McKinsey says.

How best to tap this veritable gold mine is a question being addressed by tech companies, entrepreneurs, academics and even the Obama administration. Many companies already are doing it. Ever heard of Apache Hadoop? It’s a free, open-source suite of software programs that allows precisely tailored processing of large data sets. (It was named for the creator's son's toy elephant, named Hadoop.)

The skill set necessary to effectively use Hadoop needs to be in the wheelhouse of large corporations (which may want to develop teams in-house) as well as small businesses (which are more likely to farm it out to consultants). Facebook processes billions of communications through Hadoop every day. Yahoo is a big user, too, calling it “the open source technology at the epicenter of Big Data and cloud computing.” Last year, Yahoo spun off a company called Hortonworks to further develop Hadoop, and its CEO, Eric Baldeschwieler, predicts that by 2016 half the world’s data could be trusted to Hortonworks’ care. The client list is long, including Apple, LinkedIn, Microsoft, Netflix and StumbleUpon.

Data-Driven Sales

Mollie Lombardi, research director for human capital management at the Aberdeen Group, sees rich opportunities for Big Data in the sales arena, and she uses an extremely basic personal example. “I checked into a Westin/Starwood hotel,” she says, “and the clerk said to me, ‘Welcome back; I see you were with us before—would you like to stay in the same room?’ ”

By having this information at his fingertips, the clerk was able to make a personal connection. “They had the technology to bring up that prompt to the person at the desk,” Lombardi says. “In the same way, data gathering can tell a marketing firm that I’m not going to make a purchase with a 15 percent discount—but I have a record of responding to 30 percent offers.”

Sales forces should be power users of Big Data. Assume a business manager is on the phone talking to a regular customer who says that, for $1 off per piece, he will order another 500 units. With a Big Data front end, the manager can take five or six seconds to access the customer’s history over 20 business cycles. Did the customer actually make good on his volume promises? If not, the manager is in a good position to either deny the discount or offer it conditionally on the purchase of 1,000, not 500, units.

The opportunity is there to put rich customer data in front of salespeople—and it can go well beyond a list of client kids’ birthdays to include detailed analysis of buying patterns put together from many sources in real time.

Within companies, Big Data analysis will let firms study their top-performing salespeople and gain insights into what makes them good. “We could look at graduates of X, Y and Z college and see how they’ve performed,” Lombardi says, “or study the results with people hired from competitor A versus competitor B. With information gained from sources like that, you can create a competency profile and use it to replicate the best sales hires.”

Exciting stuff, right? Not so fast. One of the problems with Big Data is that much of it is useless; according to the B2B Sales Intelligence Blog, only .01 percent of the massive amounts of data spewing forth from social networks, blogs and product reviews is helpful in discovering a buyer’s intent. Again, the key is processing and interpreting the data and gaining insight from it.