Introduction to Data Science

Open Data Science for all, or OpenDS4All, is a very good website in order to take Introduction to Data Science courses.

The first thing we want to ask is: What is data science?

Data is driving everything, and data storage is inexpensive, and parallel (compute cluster) computation is inexpensive. Can we use data and algorithms to understand phenomena, build or augment modules, build detectors, and make diagnoses? The other development worth mentioning is that we can compute with this data, for instance using cluster computation. Individually, we can get a lot by computing either on our own machines or in the cloud. What are the algorithms and data structures that are relevant here? How do we use these abilities to compute the data?

Much of science and engineering derives from physics, where we have rich predictive models. The basis of prediction tends to be simulation. A model takes certain phenomena, abstracts some characteristics and creates a computational mathematical structure based on the rest of the structure. The basis of prediction is simulation. We need enough sampling statistics and "data-first" approaches, as well as a lot of representative data. Some model simulates the reality but it's not so we have to be careful in how we are going to use these models. How do we make predictions where we don't have good models? We can make certain predictions based on the data, which we can create bad model based on the data, and get insight but with some data refinement we can do better prediction and when we + better models, we get even better predictions. This is an iterative process. For example data is cleaned with words that appear very often, to get very unique words, all related to machine learning. 

Data analytics and data engineering is the technical and data manipulation aspects of Data science. However, data science is much broader which includes storytelling and visualization. We get some data and outputs some visualization and things. In a spreadsheet, you can see pretty much all of your data, but very often in data analytics you can't and you have to use mathematical tools for data of different kinds, such as images numerical data, textual data, value sensor and it can be high dimensional data. There are many topics such as machine learning, distributed computing, distributed algorithms, and parallel computation. 

Now we can have pipelines, which are diagrams that show the ideal way of going from data to model. We have data in the left, and we have a model on the right; unfortunately; it never happens this way. The ideal way would be to get the data, extract the features of relevance press some buttons and get the model. 

The descriptive model tells the distribution of the data as a whole using normal distributions, at least as a first step. Inferential models can also be widely used for predictions. Very often; however, we have to go back from one box to another. We are going to look at some data and create these models. We want to run a simple regression model and a word cloud model.

The first concept is open in closed world. In a closed world, we assume that everything that we do not have in the data does not matter, all the inferences will be performed based on the data that I have. Databases represent all the transactions that I have made during a certain period of time, where we know everything. The open world is when we can't see the whole populations, which we are given samples or incomplete  observations. We want to predict the future or characterize the entire population. Remember, how we sample affects are model, and there are many pitfalls that can lead to biased models. We have to predict the future to characterize the entire population at times. Remember as the SEC says, "Past performance is not indicative of future results". We want to decide how many Democrats, Republicans, and Independents we can recall. For example, the model will stop working when there is a new social trend.

All the data is represented as bits but they mean different things in terms of representing images, genes, text, and data feature extraction or data wrangling. We have to understand the ways we can process images, the ways we can process genomics data and the ways that we can process text.  If we have some data we do feature extraction and we have to decide the color that is mapped in images or other manners. Sometimes we have to drop features and do feature extraction that removes certain words from the text, which is often called data preparation or data wrangling, which refer both to feature extraction and data preparation. We care about features because we use them for machine learning and to do the word clouds. Ultimately, the goal is to go form data, to information, to knowledge.

Data comes in records that can be connected to each other. If we know that certain groups of people are present, then we can predict certain types of behaviour because very often families are on the same phone plan. For example if one member changes from verizon to T-Mobile, that can very well predict a change for other members of the family.

Knowledge graphs are gaining the popularity again, since we can do a lot of stuff just by throwing a lot of data at the neural network, but some problems require more subtle knowledge, so then we have certain graphs that represent hierarchies. We should se an increasing portion of these knowledge graphs. There are also lots of weather data tracked unsuccessfully to predict weather, since climate changes are difficult to predict. 

The simplest data representation are tables, and we perform certain operations of tables like joints and lookup. A table is a matrix that contains certain types of data. We use TensorFlow to do multidimensional tables with numerical data. For example, certain taxi data can serve as a good way to predict congestion from one point to another. But that stopped working in March because the whole city got shut down. So the new data available will not work on the old model, need to look at new taxi to see how much economic activity the city has, which therefore would make it come to life. 

There is no answer to what makes data big data, but it should be too complex for humans to understand directly without a help of a machine, it does not fit into a single memory space and we need more than purely brute force algorithms to analyze them and may require multiple computers to work in parallel to process. The goal of data analytics is to create models, and to predict and perhaps intervene. For example in marketing we have data and certain combinations of factors work better from the point of view of the market campaign than others. "Show me sales region by product category". Given an observation, we can form a hypothesis and then we can experiment over the sample, "Behavioral factor F leads to higher risk of outcome O), and measure the significance compared to the null hypothesis. This is called inferential statistics. 

Big data involve acquisition, access, wrangling, integration, representation, cleaning, filtering, hypothesizing, querying, analyzing, modeling, understanding, iterating, exploring, and many ethical observations. We must also be aware of over-hyped expectations. We can't "learn everything from the bottom up" and turn the crank and pop insights. We rely on human expertise to impose models on the data and features, and deep learning can do feature selection, but we shouldn't throw away what we know. So it's a combination of experts and these machine learning algorithms that help greatly in terms of these analytics. Usually the first iteration was pretty much useless, go to the expert to see what was wrong, then correct the model, and what we provide of them was actually of high value. 

Here is the process:

What question are you answering?

What is the scope of the project? 

What data will you use?

What techniques are you going to try? 

How would you evaluate these?

What maintenance will be required?

80-90% of work comes with cataloging, wrangling, cleaning and working with experts, not working with machine learning. Data is high dimensional and hard to understand. Data science is about extracting data, and about selecting data and yes we can do clustering, classification, and recommendations of data.