Kernel Regression

 Kernels arise when we consider high-dimensional feature spaces. 

High-dimensional feature spaces can greatly extend the potential of our models. 

If I have a measurement X and restrict to operating as a weighted combination of data, then the only type of model is a straight line going through the origin. If I map this into a higher dimensional space, and associate correlation, I can model a much more complex curve. We formalize this idea here. 

I can define a ΦT(x)  that can be a factorization of polynomials. We take this approach and find W using training data, and then we want to solve the ridge regression problem to find W

We can define a matrix Φ of dimensional vectors and we can get w by solving the feature vector. 

And regression is a weighted sum of kernels. There's an interesting matrix identity. I can then rewrite my expression for d(x) as the following. The kernel is also defined here, and these elements can be written as an inner product. We define something called a kernel that represents an inner product. 

K can be computed very efficiently. For each measurement element of our vector, I can construct Φ using many values and different products of x's raised to power, provided the sum of the products is equal to q. I can write monomials in terms of a kernel, which is an inner product of 2 vectors raised to the Qth power. I can just take the inner product of u and v raised to the qth power. U transpose V involves N multiplications, and you might have to compute 100 multiplications. If P is big enough then storing Φ in memory can become a problem as a result. 

Most kernels that are used depend on the similarity as u and v. u and v are fairly similar u transpose v are large but if u and v are orthogonal then u transpose v is 0. This is a measuring the similarity of u and v. One of the most popular kernels is the Gaussian kernel, seeing how close u and v are to each other. A Gaussian kernel is an infinite dimensional Φ(x) with all order, which successfully makes it infinite dimensional. 

We take the inner product of higher dimensional feature. In the kernel factor we take the sum from i = 1 to n of αi * the kernel of (x, xi). We store and compute α in the kernel case vs w in the original formulation. By taking the sign of d(x) and having the labels equal +- 1, we can have boundaries between classes. 

We want to avoid formulation. The kernels allows us to work in higher dimensions, increasing the chance of overfitting. Choosing the kernels matter. Choosing the σ^2 affects the smoothness of the curve. One way to address the overfitting is to use cross validation to see how well your model generalizes to new data sets.