Caveats with Data Analysis

The combination of large data sets and observational data mean that data analysis activities are often at risk of drawing misleading inferences. In this blog we describe five of these dangers. Within the operations research and analytics community these problems are well understood. Unfortunately, the solution is not easy and often tempers down the enthusiasm, and to be cognizant that rather more complex models are necessary.

• Hidden nuances of data collection - Many a times the analyst is not aware of the process of data collection and the hurdles or bottlenecks of doing so. However a detailed understanding of the same renders the data analysis more holistic and improves handling observations that otherwise may lead to diametrically opposite inferences. Say for example we are studying the galactic observations from telescope to understand the distance of space objects from earth. One of the primary assumptions here will be that further the object is from earth, dimmer will it look under the telescopic lens. However there may be interstellar and intergalactic gas and dust clouds, which attenuate radiation. So it violates the primary assumption. Of course these phenomenon is known to scientists but things get more complicated or messy in situations where the data collection process are not so well understood, or, even worse, when the possibility of such events is not considered.
• Process change or Nonstationarity -Any inferences from data analysis is not valid if the there is any change in the underlying process or pattern of events. For example, a model of herbivorous animals browsing propensity on grass lands will be quite useless when the animal population declines rapidly. Any historical data on browsing will have little or no relationship to patterns after the population decline.
• Correlation Vs Causation - Its very critical to appreciate the fact that statistical correlation doesn't necessarily mean causation. High sale of umbrellas may have high correlation with number of car accidents but the cause can be either rains or snow fall. Similarly in a scatter plot we may have a positive correlation between age and height for a given range but unless we have the adequate process knowledge we would not be in a position to infer whether weight causes height to increase or vice versa !
• Pre-empting Vs Intervention - Many a times using analytics we like to prevent frauds. And to enable that we set certain thresholds either in transaction values or frequency or in absolute scale. For example in credit card transactions there are symptoms in terms of frequency and transaction values. In car insurance sector, timing of n accident claim is a good determinant for a better diligence to validate if the claim is fraudulent. However if these thresholds are some how themselves predictable then there can be interventions to avoid those detection mechanism in doing fraudulent transactions.
• Data aging - Age of data is also very critical in analytics. Real estate mortgage repayment credit history was never so bad before the sub-prime mortgage crisis hit US. However it is not easy to understand the life cycle validity of data. For example defaults in loan repayment may only happen after a minimum number of installments. For auto industry this threshold can be minimum of one to two years. For real estate the threshold can be little longer. So one has to wait that much time in the industry to understand the behavior pattern of users. However even if a strata of specific profile of consumers start behaving in a particular way after a age of 3 years since beginning of a process, there may be enough reasons that in three years the new consumers of the profile may have changed their behavior.

Moore's Law and Data science

In the last few years, there has been a burst in the amount of data that’s available. Whether its web server logs, tweet streams, online transaction records, data from sensors, government data, or some other source, the problem isn’t getting data, it’s figuring out what to do with it.

The question facing every company today even the startups and non-profist, is how to use data effectively—not just their own data, but all the data that’s available and relevant. Using data effectively requires something different from traditional statistics.

What differentiates data science from statistics is that data science is a holistic approach. We’re increasingly finding data in the wild, and data scientists are involved with gathering data, massaging it into a manageable form, finding a story/trend from it, and presenting the story to others.

Much of the data we currently work with is the direct consequence of Web 2.0, and of Moore’s Law as applied to data. The web has people spending more time online, and leaving a trail of data wherever they go. Mobile applications leave an even richer data trail, since many of them are annotated with geolocation,or involve video or audio, all of which can be mined. Point-of-sale devices and frequent-shopper’s cards make it possible to capture all of your retail transactions, not just the ones you make online. All of this data would be useless if we couldn’t store it, and that’s where Moore’s Law comes in.

The importance of Moore’s law as applied to data isn’t a theory anymore. Data expands to fill the space you have to store it. The more storage is available, the more data you will find to put into it. The data exhaust you leave behind whenever you surf the web, friend someone on Facebook, or make a purchase in your local supermarket, is all carefully collected and analyzed. Increased storage capacity demands increased sophistication in the analysis and use of that data. That’s the foundation of data science.

Data scientists combine entrepreneurship with patience, the willingness to build data products incrementally, the ability to explore, and the ability to iterate over a solution. They are inherently interdiscplinary. They can tackle all aspects of a problem, from initial data collection and data conditioning to drawing conclusions.

The future belongs to the companies who figure out how to collect and use data successfully. Google, Amazon, Facebook, and LinkedIn have all tapped into this and all other companies are trying to catch up with this at a war footing

How big is Big Data?

Big data continues to be the topic of much discussion and hype, but “big” is really a red herring. Oil companies, telecommunications companies, and other data-centric industries have had huge datasets for a long time. And as storage capacity continues to expand, today’s “big” is certainly tomorrow’s “medium” and next week’s “small.” The most meaningful definition I’ve heard: “big data” is when the size of the data itself becomes part of the problem.

We’re discussing data problems ranging from gigabytes to zetabytes of data. At some point, traditional techniques for working with data becomes inadequate. What are we trying to do with data that’s different?  We’re trying to build information platforms or dataspaces. Information platforms are similar to traditional data warehouses, but different. They are designed for capturing and understanding the data rather than for traditional need for immediate analysis and reporting. They accept variety of data formats, including the apparently incomprehensible ones, and their structures (schemas) evolve as the understanding of the data improves.

Most of the organizations that have built data platforms have found it necessary to go beyond the

relational database model. Traditional relational database systems cease to be effective at the stream of such volume(scale). Managing sharing and replication across a stack of database servers is difficult, slow and costly. The need to define a schema in advance conflicts with reality of numerous, unstructured data sources, in which you are not aware what’s important until the data is analyzed. 

Data capture and storage is only part of forming a data platform, though. Data is only useful if you can do something with it, and enormous datasets present computational problems. Machine learning is another essential tool for the data scientist. We now expect web and mobile applications to incorporate recommendation engines, and building a recommendation engine is an artificial intelligence problem.

Building statistical models plays yet another important role in any data analysis. Statistics is the “grammar of data science.” It is crucial to “making data speak coherently.” We’ve all heard the joke that eating pickles causes death, because everyone who dies has eaten pickles. That joke doesn’t work if you understand what correlation means. More to the point, it’s easy to notice that one advertisement for R in a Nutshell generated 2 percent more conversions than another. But it takes statistics to know whether this difference is significant, or just a random fluctuation. Data science isn’t just about the existence of data, or making speculation about what that data might mean; it’s about testing hypotheses and making sure that the conclusions drawn from the data are valid. Thus Statistics plays a vital role in everything from traditional business intelligence to contemporary data analytics. It isn’t superseded by newer techniques from machine learning and other disciplines; it complements them.That's where different technologies evolve in addressing the challenges in modern day data analytics including Big Data.

Big Data - A synopsis

Every day we create about few quintillion bytes of data.  90% of the data in the world today has been created in the last 2 to 3 years alone. The sudden burst in growth of data can be attributed to: posts to social media sites, digital pictures and videos, purchase transaction records, cell phone GPS signals, sensors used to gather climate information to name a few. This data is BIG DATA.

BIG DATA is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications thus it outgrows your current ability to process it, store it, and cope with it efficiently.

BIG DATA is 4D in IT spatial: Size, Acceleration, Form, Accuracy (SAFA)

Size does matter after all.

Sometimes big data is measured in terabytes, petabytes, exabyte, zettabyte or more. In real word, it's usually measured in frustration, annoyance, anxiety, and money down the drain.

The challenges include capture, curation, storage, search, sharing, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, legal citations, combat crime, and determine real-time roadway traffic conditions

Acceleration

For time critical processes such as fraud detection in trade events, predict customer churn etc. BIG DATA must be used as it flows into your enterprise in order to maximize it's value. It's just not a race against time rather to derive potential insights that provides innovative ways of doing things.

Form

BIG DATA is varied type of data - structured and unstructured data such as text, video, audio, sensor data, click streams, log files and much more. New insights are found when these data types are analyzed together.

Accuracy

1 in 3 business leader do not trust the information they use to make decisions. How can one act upon information that they don't trust. Establishing trust in BIG DATA presents a huge challenge and the variety and number of source grows.

BIG DATA is more than simply a matter of size; it is opportunity to find insights in new and emerging types of data and content.

Challenges of Big Data

What is Big Data ? Starting from scientists struggling to analyze massive amounts of genetic data, as they did in the Human Genome Project a decade ago, or astronomical data, such as the survey of more than 930,000 galaxies undertaken by the Sloan Digital Sky Survey, Big Data has blossomed into a constellation of computer science approaches to handling, visualizing and blending together "big" sets of data.

However following are the challenges for Big Data

1. Data storage implications -  People are creating staggering amounts of data just through the course of their daily lives, and the volume is growing even larger from data flowing from "machines and implanted sensors in oceans, in the soil, in pallets of products, in gambling casino chips, in pet collars, and countless other places." Starting from switching on your cable TV in the morning based on which genre of channel you like, to making a phone call (GPRS location, talk time and calling profile), to filing up gas with your credit card (trending of gas usage based on days of the week and the time of the day) to browsing the net to either google or tweet is generating terabytes or zeta bytes of data. Imagining the world getting smaller and smaller every day and large multinational expanding their presence rapidly across the world expanding their customers in geometric progression, storage of data is a big challenge. 
2. Technology - The right technologies and methods to store, map and process such huge streams of data are still evolving. There are number of tools like Hadoop for mapping global data and storing data into localized nodes. Languages (like Hive) are also developed or reused ( like SQL) to query and fetch the relevant data from file systems. And there are statistical tools like 'R' which are used to analyze such data sets. However none of these are matured systems and without certain constraints. It will take some time for the technologies to be seamless and acquiring the capability to do the needful end to end for Big Data.  
3. Cost vs return - Industry experts are still grappling with the cost benefit analysis of Big Data. Certain practitioners believe that social networking data as was thought to be useful in driving creative marketing and hence impacting the top line revenue has not been that successful in past. However optimists are of the opinion that though the intent was there in the past there was a lack of tools and technology to achieve that. With the IT giants lie IBM and others in the field now, Big Data certainly is very promising and will be bigger wave than either Y2K or even the IT boom that the world has witnessed in the last one and half decade.
4. Privacy - The biggest worry is the privacy concern. As an article from CNN noted, "it has become cheap and easy to pry into the lives of others at the same time that protecting our own lives has become time-consuming and expensive." Andrew Rowland, global head of database engineering at UBS, said there is a privacy and a relevancy element surrounding data: “You might have bought ladies underwear online once, but do you want to be presented with it every day? Do people want every bit of data assessed for them?”
5. Sampling - Big data contradicts the basis of statistical sampling. This is a dichotomy and challenges the premise of age old concept of taking a fraction of homogenous population or stratified portion of the population to predict and infer characteristics and behavior of the population. Big data takes into count the entire data set and analyzes it. Unless the ROI is disproportionally high, Big data will face challenges from the businesses in near future.
6. Data accuracy - The popular cliche 'garbage in, garbage out' is valid here also. Unless the data is generated directly by the end user and fed directly into data storage systems there is a huge chance that the data is not correct. And if improper data is used for analysis it can create huge lost opportunity and resource not to mention of any detrimental impact on marketing if any.

Business users need to properly understand and make sense of the data their business produces. The big data paradigm must be made useful to business; rather than being just a corporate conversation turning into another FAD.

BI vs. BA

What’s the difference between Business Analytics and Business Intelligence? The correct answer is: everybody has an opinion, but no one is wrong nor there is just one perfect answer.

Everybody has a different notion of what ANY particular term associated with analytics means.

However "business intelligence” is an umbrella term including data warehousing, business intelligence, enterprise information management, enterprise performance management, analytic applications, and governance, risk, and compliance.

“Business analytics” on the other hand indicates some level of vertical/horizontal domain knowledge tied with statistical or predictive analytics.

At the end of the day, there are two aspects worth differentiating:

1. Analysis and Inference — the need to get the most value out of information. This need hasn’t really changed in over fifty years (although the increasing complexity of the world economy means new concepts like unstructured data, real-time data, Big Data etc.). And the majority of real issues that stop us from getting value out of information (information culture, politics, lack of analytic competence, lack of storage, data noise etc.) haven’t changed in decades either.
2. The second is the IT aspect of BI — what technology is used to help provide the business need. This obviously does change over time. There are different analysis tools like SPSS, SAS, R, Minitab and hosts of data storage mechanisms in warehousing or file systems like Cognos, Oracle, Hadoop etc. 

The problems in nomenclature typically arise because “business intelligence” is commonly used to refer both of these, according to the context, thus confusing everyone.

At the end of the date, nobody important cares what this stuff is called. If you’re in charge of a project, what matters is working out the best way to leverage the information opportunity in your organization, and putting in place appropriate technology to meet that business need — and you can call that process whatever you like: it won’t make any difference…