Data Science Part 14: Decision Theory, Linear Regression, Clustering

The decision theory is a more generalized problem in data science. Here, we simple want to minimize the loss function. We've seen the square error cost function and the absolute error function, denoted by equations

and 

,

respectively. The square error loss function, emphasizes/accentuates larger errors, where the smaller error receives more weight inside of the absolute error function. The Squared error loss function is Bayesian in nature, with conditional expectation Yhat = E[Y|X]. The absolute error leads to median estimator; typically viewed as more robust. These are l1 and l2 functions, respectively. L1 is better dealing with outliers, while L2 yields more of the expected value. Consider a dataset Y = {2, 1, 1.5, 2, 17}. We compute error as 

and attempt to compute the derivative of this error. We then take the derivative as 

We set the derivative to 0 to find the β that minimizes the error, and β = 4.7 from this. This could be proved as expected value since the average of Y = 4.7. To compute the error for l1 we want to minimize the function

which happens when  β=2, the median of the set. This is because derivative is 1 when yn > β else -1.

For a univariate model,  β can be optimized as 

and this is in the scope of univariate linear regression. The first numerator is <x,y> and the denominator is <x,x>. In multivariate linear regression, orthogonal means <xj, xk> = 0 for j != k. As a result we can show that βi = <xi, yi>/<xi,xi>. So for multivariate linear regression we can solve the problem by orthogonalizing first through Graham Schmidt orthogonalization. 

Now, let's discuss clustering. We want to group data into “clusters” that seem related. There is a degree of similarity between different clusters. The algorithmic approach is iterative; we move points between clusters, then recalculate cluster centers.

In the K means algorithm, we initialize K cluster centers. We will select K points among training data. Until convergence, we associate each training datum with nearest cluster center, then recompute cluster centers as average of training data in cluster. We Map datum xn to cluster C(n)=k to representation level rk,

In equation terms it is denoted as 

We take the cluster center by computing average of all cluster points in Kmeans. 

Here's the pseudocode: 

## K-Means Clustering 

1. Choose the number of clusters(K) and obtain the data points 

2. Place the centroids c_1, c_2, ..... c_k randomly 

3. Repeat steps 4 and 5 until convergence or until the end of a fixed number of iterations

4. for each data point x_i:

       - find the nearest centroid(c_1, c_2 .. c_k) 

       - assign the point to that cluster 

5. for each cluster j = 1..k

       - new centroid = mean of all points assigned to that cluster

6. End 

Now let's point back to the linear regression point. The Xj in linear regression can be actual inputs, function of inputs, polynomials of inputs, codes of classes, and interactions between variables. X points can be in multiple dimensions as well, as each column is enumerated as a different dimension. An example is considering financial prediction where Yn is the % change in stock price. Xn1 can be company earnings, Xn2 could be Book value, Xn3 could be price momentum, etc.

We want to predict future returns (% price change) from data. We get the x variables for each stock. We can use least squares to calculate parameters β, and subsequently apply parameters for linear model to predict future returns.