Data Science PT 1: Introduction, Applications, and Curve fitting.

Data science is the extraction of knowledge from data in various forms, regardless of structured or unstructured. We can extract a bigger product/utility/benefit/etc. 

Now, let's get into the applications of Data Science. One such application is click prediction. Using ads online and clicks, we can track various data related to each ad and create a better prediction which will result in more ad revenue and happier customers for retention. Another application here is speech recognition, where we recently shifted from hidden Markov models to training lots of data. Alternatively, we can use data science to predict mortgage default, so the bank won't lose a lot of money. Similarly we can do the same for other insurance payments, using mainly binary classification. We can also use data science in financial prediction to try to design a portfolio that goes up in low volatility. Data science is also used in Games where in the Game of Go AlphaGo beat the champion Go player who is a human. We can also identify handwriting where the post office wants to recognize zip codes, with a nearest neighbors approach. A final example of data science that I will mention in the article is Elon Musk and his self-driving cars. 

The next topic that I want to describe here is polynomial curve-fitting. Here we have function tn with target variable t and input variable x and we want to use this to predict future target variables. We want to find an order N polynomial that best explains t. We could use a Taylor Series in turn to approximate the function. Perhaps a sparse function in the Fourier basis is better so far. A sparse function means that a small number of nonzero elements describe it well. From here we can predict that y(x, w) = t' = t(x) = w o x. Functions are below. We want only "squared" errors because higher powers will bring out outliers more. We also want to have a Gaussian distribution of the squared error. 

We want an appropriate order. Too much order will overfit. Too little order won't be accurate enough. There has to be a fine line. To estimate a good model M, we will use Test data. We use training data to compute optimal weights W, and test data to show how well w explains the remaining data. We need a sweet spot minimizing test error, as well as a reasonable tradeoff for training error. With more polynomials we need more columns. Usually we want to see the fitting error to be equal to (or maybe even slightly less than the standard deviation. For example if the variance is 0.01 then we want the error to be around 0.1 sufficient enough to have a good fit but not small enough to fail with the test data.