In linear classification, the data X takes values in class G={1,…,K}. We want to predict G(X).
We want to create a linear/affine function
The goal is to try to predict the function that maximizes fk(X). In quadratic boundaries, we want to add cross terms (X1X2, X1^2, X2^2) in space.
Another goal is to define K indicator variables, where
We form K variables into vector, Y=(Y1,…,YK). The N training instances form a N×K indicator matrix. Again, we yield a final pseudo inverse function of
The interpretation of this is that fk(X) resembles Pr(G(X)=k|X) due to E(Indicator) = Probability. As N increases we will add more basis expansions/polynomial terms and run regression on those. In general we want a K - 1 Polynomial, where K is the number of classes.
The following is a Bayesian approach to classification:
Consider fk(X) being posterior densities, fk(X) = f(X|k).
In the following problem, we want to attempt to derive a Bayesian Classification Approach for 2 classes governed by the distributions fblue=0.5N(-1,1)+0.5N(2,1) and fred=N(0,1). There are 2 Gaussian components for the blue class.
Possible posterior models are Gaussian densities (linear discriminant analysis), Gaussian Mixtures (sum of several Gaussian Densities), Nonparametric Densities, and Naïve Bayes (assumes that the multivariate pdf is the product of the p individual distributions). The covariance of 2 variables is the degree in which they go together. It's very similar to correlation.
For multivariate Gaussian density, we select cluster mean 𝜇𝑘 and covariance matrix Σk, with its computation rules shown above. fk(x) here an be defined as
We can therefore assume the same covariance matrix for all classes. Each class has Gaussian centered around µk. We can assume an elliptical structure for Gaussian component, heavily based on the covariance Σ.
Linear discriminant analysis(LDA) helps to set a linear decision boundary between 2 classes. We can show that
LDA uses the data to create a new axis and it uses the axis to maximize the separation between the 2 categories.
We want to maximize the mean difference and minimize the variation difference; that is, maximize the term
.
We want to choose the k that maximizes 𝛿k(X), where
. It's pretty closely related to linear regression approach. It tries to maximize the separation of known categories. While LDA assumes same Gaussian, Quadratic Discriminant Analysis allows different Gaussians. Aka, the covariance matrices are different. Each class has a different elliptical set, which sets the decision boundaries to be quadratic as a result. In general, LDA and QDA have good performance, yielding robust/stable boundaries. QDA requires more parameters than LDA and needs a somewhat larger N to work well. QDA also has lower error rate because the assumptions fit the data.
Left shows LDA and right shows QDA.
Comments
Post a Comment