The Year of BIG DATA

Happy New Year!

"Let this coming year be better than all the others. Vow to do some of the things you've always wanted to do but couldn't find the time. Call up a forgotten friend. Drop an old grudge, and replace it with some pleasant memories. Vow not to make a promise you don't think you can keep. Walk tall, and smile more. You'll look ten years younger." - Ann Landers

Unlike the hundreds of thousands of people who braved the cold and thronged to the crossroads of the world - the heart of Times Square in New York City - to watch the ball drop at the stroke of midnight to usher in 2012, many of us were quite content to sip champagne and watch the events unfold on television from the warmth of our living rooms. Here is an excerpt from an editorial written by me that appeared in the January 2012 issue of The Quality Islander.

The New Year brings with it anticipation of new opportunities. In order to capitalize on these opportunities we need to be able to identify business trends and develop strategies that will deliver products with just the right features, reduce operational costs and improve service to our customers. How do we do this? Very simple! Understand customer behavior. This is where the ability to comprehend "Big Data” becomes critical for businesses to stay competitive and profitable.  IBM defines big data as the "2.5 quintillion (2.5 followed by 18 zeroes) bytes of data we create every day from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and cell phone GPS signals, to name a few. Ninety percent of this data has been created in the last two years alone."

I have spent much of last year working with big data. Until a year ago it was only the Internet companies who were interested in analyzing big data comprised mostly of click stream information to better understand the visitors to their web sites so that they can deliver advertisements with personalized contents. This type of data analysis, called data mining, is now increasingly being used by many companies including utilities, media companies, hospitals, banks and retailers. For example, utilities have started to mine data to predict customer demand for electric power and monitor the efficiency of the distribution grid. Media companies that deliver programming via the Internet or cable are now able to monitor in real-time the demography of their customers watching their content. This information can be used to develop customized programming at the household level and for targeted marketing.

Unfortunately, as big data grows there is a continuous erosion of our privacy because the results of the analyses of big data, called business intelligence (BI), are increasingly being used to make decisions that affect our everyday lives. BI used by hospitals directly determines the quality and type of medical care delivered to patients. BI used by banks determines the cost at which a customer can borrow money, which implies that not only must big data be clean but also the right metrics and models must be used to analyze big data.  As the volume of big data increases the complexity of the analyses can afford to decrease, according to a Gartner report which says, “Today, data mining is really about building sophisticated models with not very much data. Now, big data gives you huge volumes of data which means that you don’t need as sophisticated a model anymore.”  In other words, the cleanliness, consistency and integrity of data will be at least as important as the level of complexity of the analytical models.  In addition, non-traditional types of databases will be needed to store, classify and quickly access unstructured data from social media.

There are many new technologies scheduled for release this year that will help us better manage and understand big data. In order to stay competitive, companies will do well to carefully evaluate these new technologies for adaptability and scalability to make sure that the technology they choose is aligned with the types of data they collect and the needs of their customers.

Comments

[Bruce P]

Most companies have abandoned long-term initiatives for short-term profits. As long as this trend continues and companies are able to keep their shareholders happy there is little incentive to make process changes (like Agile) that will turn the company on its head.