Data Science Part 17: Logistic regression, basis expansions

Let's first discuss the concept of logistic regression. We want to model posteriors linearly. From class 0 to K - 1, the Equation to do this is

We can show that

However, the hard part of logistic regression is learning the parameters. Logistic regression predicts if something is true or false, in contrast to something continuous, like size. We have to see if a variable's effect on a prediction is significantly different from 0. If not, then this means that the variable is not affecting the prediction. It uses "maximum likelihood", and can classify the y value binarily based on the x value. To iterate, calculate the likelihood of all events in a chart. Based on the multiplication of all these likelihood, shift the line and calculate the new likelihood of data points. 

Here, we calculate the log(odds of obesity). The logit function is log(p/(1-p)).

First, come up with a linear regression line. 

The sigmoid function has asymptotes at 1 and 0 and middle at 0.5. We define it as 

The logistic function cost function is 

cost(hθ(x), y) = {-log(hθ(x)) if y = 1 and -log(1 - hθ(x) if y = 0} and overall cost is

we can combine everything into the following effective equation J(θ):

 We want to repeatedly update each parameter through gradient descent until we get a good fit. 

Now, time to discuss basis expansions.

So far, we have discussed linear models quite a bit, and we can define linear models by a first-order Taylor approximation. It's hard to fit more complex functions when N small or P large. In basis expansions, we want to augment/replace inputs. We begin with a vector of inputs (X1, X2, ... Xp). We then augment/replace inputs with transformations of X. The linear basis expansion of f(x) is

Here are some example basis expansions: 

1. hm(X)=Xm  is the original linear model

2. hm(X)=(Xj)^2 or XjXk second (or higher) order Taylor series. When the number of terms grow, we need to regularize.

3. Piecewise Polynomials/Splines. This allows us to augment the inputs with polynomial terms to achieve higher order Taylor expansions.

This concept of a family of transformations that can fit together to capture general shapes is called a basis expansion. 

We can control complexity using variable selection or regularization. These piecewise functions help to fit for local structure, and enforce continuity. Constraints reduces degrees of freedom in piecewise polynomials. We divide x into interval and represent f(x) with separate basis functions in each interval. We can also use cubic splines, multi-dimensional splines/etc. The big idea is to augment/replace the vector of inputs with additional variables, then use linear models in the new space of derived input features. It's generally regarded as the best way to capture non-linear relationships.

We can use basis functions in the realm of financial data as well. Linear model for future prices may be restrictive. Polynomial expansions is highly sensitive to noise in the data and overfit violently in every example. Binning is cutting the range of the predictor variable into equally sized intervals. However, it produced discontinuous functions.