Data Science Part 15: Subset selection, shrinkage, decision trees

 

As good as least squares is, it does have some significant problems that need to be solved. The Least squares method often outputs entirely nonzero vector, and maybe for some usable results. Least squares doesn’t use possible knowledge about structure of β. The multiplication of pseudoinverse ignores its structure (sparse β, discrete valued β, non-negative β, …). For example if we want to see if a phone user is active or inactive, least squares won't work because the using the equation Y=HX+Noise; X comprised of {-1,0 (inactive),+1},

H+Y contains elements not in {-1,0,+1}; however, we want output to exclusively be {-1, 0, 1}. Therefore, we want to try to select a subset of columns of matrix X that approximate the Y value reasonably well. Various techniques provide faster runtime, and may approach best solution but not find it. They require random variables of X to be identically and independently distributed. For example, in a medical setting, the columns of X are measurements relating to patients, and Y is the medical outcome that we're trying to predict. 

We want to understand the relations that occur with the most important factors, and subset selection helps save more money but making a dataset much simpler for a human to work with. We often want to estimate the β vector while minimizing the mean squared error (MSE). We want to optimize the conditional expectation, 

We can write the summation of β as the sum of all possible β'. We take a weighted average over all possible β 0 , where the weights are posterior probabilities conditioned on X and Y .We want to predict the best combination to use in regards to this model. We mainly will have nonzero values since the function is continuous. If we select s columns out of p total columns as a subset, we have 

different possible sub columns choices. There are so many, so we just have to pick a "good enough" result. 

Now onto the topic of shrinkage. Subset selection throws out some columns that may have large errors. Shrinkage methods modify β more subtly by shrinking elements of β back to 0. Ridge Regression will penalize a large β.

Various algorithms implement LASSO, which helps makes the process faster. To estimate sparse β, we need to first realize that LASSO can drive many coefficients to zero. We can also perform graph reduction as a way to simplify problems. Zero measurement implies sparse and real valued β. Matching measurements imply that the non-zeros in X are real numbers. 

Bason on this diagram, we can base the last value of X to be 3. Solving these sparse matrices is similar to solving a sudoku puzzle. Again, we try to make β as close to zero as possible to reduce the complexity. 

Ridge regression prevents overfitting and helping make the model a lot more generalized. We can trade off doing worse during training to have more generalized and better testing results. The penalizing terms in ridge regression helps to reduce the weights and biases. Ridge regression tries to work by increasing the bias to improve/decrease variance. In this way, the model has become a lot less sensitive to changes. As α increases, the line becomes more and more horizontal. 

In contrast, in LASSO the λ (always > 0??) value is similar, yielding

adding a squared l2 norm with a l2 norm. 

β is also independent of X. The maximum a posteriori value is 

As a result, we want to try to maximize f(β, X, y) because it will have the greatest Gaussian Probability.

The following system of equation steps ensues: 

We can also model each of the scalar entries as

Remember the log function constantly travels either one direction or another, so we can conclude that 

Per Wikipedia, a decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm that only contains conditional control statements. The branches are the decisions(X) and the leaves are the actions (Y). Decision trees can perform classification and crude regression, as there are numerical values at the tree leaves. There are actual functions at leaves. Each internal node of the tree represents an attribute, and each leaf node a class label. Decision trees are suitable for handling heterogeneous data, because they can work directly on categorical data types.

We want leaves to explain the data effectively. We can perfectly explain training data for one leaf per datum. We want to penalize structure to avoid overfitting.