Challenges of Big Data

What is Big Data ? Starting from scientists struggling to analyze massive amounts of genetic data, as they did in the Human Genome Project a decade ago, or astronomical data, such as the survey of more than 930,000 galaxies undertaken by the Sloan Digital Sky Survey, Big Data has blossomed into a constellation of computer science approaches to handling, visualizing and blending together "big" sets of data.

However following are the challenges for Big Data

1. Data storage implications -  People are creating staggering amounts of data just through the course of their daily lives, and the volume is growing even larger from data flowing from "machines and implanted sensors in oceans, in the soil, in pallets of products, in gambling casino chips, in pet collars, and countless other places." Starting from switching on your cable TV in the morning based on which genre of channel you like, to making a phone call (GPRS location, talk time and calling profile), to filing up gas with your credit card (trending of gas usage based on days of the week and the time of the day) to browsing the net to either google or tweet is generating terabytes or zeta bytes of data. Imagining the world getting smaller and smaller every day and large multinational expanding their presence rapidly across the world expanding their customers in geometric progression, storage of data is a big challenge. 
2. Technology - The right technologies and methods to store, map and process such huge streams of data are still evolving. There are number of tools like Hadoop for mapping global data and storing data into localized nodes. Languages (like Hive) are also developed or reused ( like SQL) to query and fetch the relevant data from file systems. And there are statistical tools like 'R' which are used to analyze such data sets. However none of these are matured systems and without certain constraints. It will take some time for the technologies to be seamless and acquiring the capability to do the needful end to end for Big Data.  
3. Cost vs return - Industry experts are still grappling with the cost benefit analysis of Big Data. Certain practitioners believe that social networking data as was thought to be useful in driving creative marketing and hence impacting the top line revenue has not been that successful in past. However optimists are of the opinion that though the intent was there in the past there was a lack of tools and technology to achieve that. With the IT giants lie IBM and others in the field now, Big Data certainly is very promising and will be bigger wave than either Y2K or even the IT boom that the world has witnessed in the last one and half decade.
4. Privacy - The biggest worry is the privacy concern. As an article from CNN noted, "it has become cheap and easy to pry into the lives of others at the same time that protecting our own lives has become time-consuming and expensive." Andrew Rowland, global head of database engineering at UBS, said there is a privacy and a relevancy element surrounding data: “You might have bought ladies underwear online once, but do you want to be presented with it every day? Do people want every bit of data assessed for them?”
5. Sampling - Big data contradicts the basis of statistical sampling. This is a dichotomy and challenges the premise of age old concept of taking a fraction of homogenous population or stratified portion of the population to predict and infer characteristics and behavior of the population. Big data takes into count the entire data set and analyzes it. Unless the ROI is disproportionally high, Big data will face challenges from the businesses in near future.
6. Data accuracy - The popular cliche 'garbage in, garbage out' is valid here also. Unless the data is generated directly by the end user and fed directly into data storage systems there is a huge chance that the data is not correct. And if improper data is used for analysis it can create huge lost opportunity and resource not to mention of any detrimental impact on marketing if any.

Business users need to properly understand and make sense of the data their business produces. The big data paradigm must be made useful to business; rather than being just a corporate conversation turning into another FAD.

BI vs. BA

What’s the difference between Business Analytics and Business Intelligence? The correct answer is: everybody has an opinion, but no one is wrong nor there is just one perfect answer.

Everybody has a different notion of what ANY particular term associated with analytics means.

However "business intelligence” is an umbrella term including data warehousing, business intelligence, enterprise information management, enterprise performance management, analytic applications, and governance, risk, and compliance.

“Business analytics” on the other hand indicates some level of vertical/horizontal domain knowledge tied with statistical or predictive analytics.

At the end of the day, there are two aspects worth differentiating:

1. Analysis and Inference — the need to get the most value out of information. This need hasn’t really changed in over fifty years (although the increasing complexity of the world economy means new concepts like unstructured data, real-time data, Big Data etc.). And the majority of real issues that stop us from getting value out of information (information culture, politics, lack of analytic competence, lack of storage, data noise etc.) haven’t changed in decades either.
2. The second is the IT aspect of BI — what technology is used to help provide the business need. This obviously does change over time. There are different analysis tools like SPSS, SAS, R, Minitab and hosts of data storage mechanisms in warehousing or file systems like Cognos, Oracle, Hadoop etc. 

The problems in nomenclature typically arise because “business intelligence” is commonly used to refer both of these, according to the context, thus confusing everyone.

At the end of the date, nobody important cares what this stuff is called. If you’re in charge of a project, what matters is working out the best way to leverage the information opportunity in your organization, and putting in place appropriate technology to meet that business need — and you can call that process whatever you like: it won’t make any difference…