Caveats with Data Analysis

The combination of large data sets and observational data mean that data analysis activities are often at risk of drawing misleading inferences. In this blog we describe five of these dangers. Within the operations research and analytics community these problems are well understood. Unfortunately, the solution is not easy and often tempers down the enthusiasm, and to be cognizant that rather more complex models are necessary.

• Hidden nuances of data collection - Many a times the analyst is not aware of the process of data collection and the hurdles or bottlenecks of doing so. However a detailed understanding of the same renders the data analysis more holistic and improves handling observations that otherwise may lead to diametrically opposite inferences. Say for example we are studying the galactic observations from telescope to understand the distance of space objects from earth. One of the primary assumptions here will be that further the object is from earth, dimmer will it look under the telescopic lens. However there may be interstellar and intergalactic gas and dust clouds, which attenuate radiation. So it violates the primary assumption. Of course these phenomenon is known to scientists but things get more complicated or messy in situations where the data collection process are not so well understood, or, even worse, when the possibility of such events is not considered.
• Process change or Nonstationarity -Any inferences from data analysis is not valid if the there is any change in the underlying process or pattern of events. For example, a model of herbivorous animals browsing propensity on grass lands will be quite useless when the animal population declines rapidly. Any historical data on browsing will have little or no relationship to patterns after the population decline.
• Correlation Vs Causation - Its very critical to appreciate the fact that statistical correlation doesn't necessarily mean causation. High sale of umbrellas may have high correlation with number of car accidents but the cause can be either rains or snow fall. Similarly in a scatter plot we may have a positive correlation between age and height for a given range but unless we have the adequate process knowledge we would not be in a position to infer whether weight causes height to increase or vice versa !
• Pre-empting Vs Intervention - Many a times using analytics we like to prevent frauds. And to enable that we set certain thresholds either in transaction values or frequency or in absolute scale. For example in credit card transactions there are symptoms in terms of frequency and transaction values. In car insurance sector, timing of n accident claim is a good determinant for a better diligence to validate if the claim is fraudulent. However if these thresholds are some how themselves predictable then there can be interventions to avoid those detection mechanism in doing fraudulent transactions.
• Data aging - Age of data is also very critical in analytics. Real estate mortgage repayment credit history was never so bad before the sub-prime mortgage crisis hit US. However it is not easy to understand the life cycle validity of data. For example defaults in loan repayment may only happen after a minimum number of installments. For auto industry this threshold can be minimum of one to two years. For real estate the threshold can be little longer. So one has to wait that much time in the industry to understand the behavior pattern of users. However even if a strata of specific profile of consumers start behaving in a particular way after a age of 3 years since beginning of a process, there may be enough reasons that in three years the new consumers of the profile may have changed their behavior.

Moore's Law and Data science

In the last few years, there has been a burst in the amount of data that’s available. Whether its web server logs, tweet streams, online transaction records, data from sensors, government data, or some other source, the problem isn’t getting data, it’s figuring out what to do with it.

The question facing every company today even the startups and non-profist, is how to use data effectively—not just their own data, but all the data that’s available and relevant. Using data effectively requires something different from traditional statistics.

What differentiates data science from statistics is that data science is a holistic approach. We’re increasingly finding data in the wild, and data scientists are involved with gathering data, massaging it into a manageable form, finding a story/trend from it, and presenting the story to others.

Much of the data we currently work with is the direct consequence of Web 2.0, and of Moore’s Law as applied to data. The web has people spending more time online, and leaving a trail of data wherever they go. Mobile applications leave an even richer data trail, since many of them are annotated with geolocation,or involve video or audio, all of which can be mined. Point-of-sale devices and frequent-shopper’s cards make it possible to capture all of your retail transactions, not just the ones you make online. All of this data would be useless if we couldn’t store it, and that’s where Moore’s Law comes in.

The importance of Moore’s law as applied to data isn’t a theory anymore. Data expands to fill the space you have to store it. The more storage is available, the more data you will find to put into it. The data exhaust you leave behind whenever you surf the web, friend someone on Facebook, or make a purchase in your local supermarket, is all carefully collected and analyzed. Increased storage capacity demands increased sophistication in the analysis and use of that data. That’s the foundation of data science.

Data scientists combine entrepreneurship with patience, the willingness to build data products incrementally, the ability to explore, and the ability to iterate over a solution. They are inherently interdiscplinary. They can tackle all aspects of a problem, from initial data collection and data conditioning to drawing conclusions.

The future belongs to the companies who figure out how to collect and use data successfully. Google, Amazon, Facebook, and LinkedIn have all tapped into this and all other companies are trying to catch up with this at a war footing

How big is Big Data?

Big data continues to be the topic of much discussion and hype, but “big” is really a red herring. Oil companies, telecommunications companies, and other data-centric industries have had huge datasets for a long time. And as storage capacity continues to expand, today’s “big” is certainly tomorrow’s “medium” and next week’s “small.” The most meaningful definition I’ve heard: “big data” is when the size of the data itself becomes part of the problem.

We’re discussing data problems ranging from gigabytes to zetabytes of data. At some point, traditional techniques for working with data becomes inadequate. What are we trying to do with data that’s different?  We’re trying to build information platforms or dataspaces. Information platforms are similar to traditional data warehouses, but different. They are designed for capturing and understanding the data rather than for traditional need for immediate analysis and reporting. They accept variety of data formats, including the apparently incomprehensible ones, and their structures (schemas) evolve as the understanding of the data improves.

Most of the organizations that have built data platforms have found it necessary to go beyond the

relational database model. Traditional relational database systems cease to be effective at the stream of such volume(scale). Managing sharing and replication across a stack of database servers is difficult, slow and costly. The need to define a schema in advance conflicts with reality of numerous, unstructured data sources, in which you are not aware what’s important until the data is analyzed. 

Data capture and storage is only part of forming a data platform, though. Data is only useful if you can do something with it, and enormous datasets present computational problems. Machine learning is another essential tool for the data scientist. We now expect web and mobile applications to incorporate recommendation engines, and building a recommendation engine is an artificial intelligence problem.

Building statistical models plays yet another important role in any data analysis. Statistics is the “grammar of data science.” It is crucial to “making data speak coherently.” We’ve all heard the joke that eating pickles causes death, because everyone who dies has eaten pickles. That joke doesn’t work if you understand what correlation means. More to the point, it’s easy to notice that one advertisement for R in a Nutshell generated 2 percent more conversions than another. But it takes statistics to know whether this difference is significant, or just a random fluctuation. Data science isn’t just about the existence of data, or making speculation about what that data might mean; it’s about testing hypotheses and making sure that the conclusions drawn from the data are valid. Thus Statistics plays a vital role in everything from traditional business intelligence to contemporary data analytics. It isn’t superseded by newer techniques from machine learning and other disciplines; it complements them.That's where different technologies evolve in addressing the challenges in modern day data analytics including Big Data.

Big Data - A synopsis

Every day we create about few quintillion bytes of data.  90% of the data in the world today has been created in the last 2 to 3 years alone. The sudden burst in growth of data can be attributed to: posts to social media sites, digital pictures and videos, purchase transaction records, cell phone GPS signals, sensors used to gather climate information to name a few. This data is BIG DATA.

BIG DATA is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications thus it outgrows your current ability to process it, store it, and cope with it efficiently.

BIG DATA is 4D in IT spatial: Size, Acceleration, Form, Accuracy (SAFA)

Size does matter after all.

Sometimes big data is measured in terabytes, petabytes, exabyte, zettabyte or more. In real word, it's usually measured in frustration, annoyance, anxiety, and money down the drain.

The challenges include capture, curation, storage, search, sharing, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, legal citations, combat crime, and determine real-time roadway traffic conditions

Acceleration

For time critical processes such as fraud detection in trade events, predict customer churn etc. BIG DATA must be used as it flows into your enterprise in order to maximize it's value. It's just not a race against time rather to derive potential insights that provides innovative ways of doing things.

Form

BIG DATA is varied type of data - structured and unstructured data such as text, video, audio, sensor data, click streams, log files and much more. New insights are found when these data types are analyzed together.

Accuracy

1 in 3 business leader do not trust the information they use to make decisions. How can one act upon information that they don't trust. Establishing trust in BIG DATA presents a huge challenge and the variety and number of source grows.

BIG DATA is more than simply a matter of size; it is opportunity to find insights in new and emerging types of data and content.